How to create an API with Symfony 4 and JWT

Today we’re going to create a Symfony 4 API web app from scratch – I’ll walk you through all the steps, so by the end of this tutorial, you should be able to create, configure and run a web app with API endpoints and protected with JWT authentication.

Also, I’ve uploaded all the source code here so you can follow through the tutorial or you can download the code and play with it while you read.

1. Docker

To set our development environment, we’ll use Docker – you probably already know by now how much I love Docker 🙂

Let’s start by creating a docker-compose.yaml file with php7, mysql for database and nginx for the webserver.

#docker-compose.yaml
version: "3.1"

volumes:
    db-data:

services:
    mysql:
      image: mysql:5.6
      container_name: ${PROJECT_NAME}-mysql
      working_dir: /application
      volumes:
        - db-data:/application
      environment:
        - MYSQL_ROOT_PASSWORD=docker_root
        - MYSQL_DATABASE=sf4_db
        - MYSQL_USER=sf4_user
        - MYSQL_PASSWORD=sf4_pw
      ports:
        - "8306:3306"

    webserver:
      image: nginx:alpine
      container_name: ${PROJECT_NAME}-webserver
      working_dir: /application
      volumes:
        - .:/application
        - ./docker/nginx/nginx.conf:/etc/nginx/conf.d/default.conf
      ports:
        - "8000:80"

    php-fpm:
      build: docker/php-fpm
      container_name: ${PROJECT_NAME}-php-fpm
      working_dir: /application
      volumes:
        - .:/application
        - ./docker/php-fpm/php-ini-overrides.ini:/etc/php/7.2/fpm/conf.d/99-overrides.ini
      environment:
        XDEBUG_CONFIG: "remote_host=${localIp}"

Also, don’t forget the .env file with your PROJECT_NAME variable. On this ocasion, for the php-fpm and the nginx, I’m pointing to docker folder, so I can override the nginx.conf file and the php-fpm Dockerfile with special configuration, such as xdebug.

Link to docker/nginx/nginx.conf file
Link to docker/php-fpm/Dockerfile
Link to docker/php-fpm/php-ini-overrides.ini

Once we’re ready, we can build and run.

docker-compose build
docker-compose up -d

2. Creating a Symfony project

First, let’s go into the bash

docker-compose exec php-fpm bash

Let’s create a symfony 4 project

#inside php-fpm bash
composer create-project symfony/website-skeleton symfony

Clean up

#inside php-fpm bash
mv /application/symfony/* /application
mv /application/symfony/.* /application
rm -Rf /application/symfony

More on how to create a symfony 4 application with docker here

If we open the browser http://localhost:8000/ we should see the Symfony welcome page. So now that we have a symfony 4 app up and running, let’s start putting stuff into it!

3. Mapping our User in the database

Inside the php-fpm bash, let’s start with installing the FOSUserBundle to have a User base entity we can relate to.

#inside php-fpm bash
composer require friendsofsymfony/user-bundle "~2.0"
Note: you'll probably get the error "The child node "db_driver" at path "fos_user" must be configured." Don't panic, this is unfortunately normal. Reason being is it's trying to clear the cache before configuration is correct.

3.1 Configuriation

#config/services.yaml

# FOS user config
fos_user:
    db_driver:      orm # other valid values are 'mongodb', 'couchdb' and 'propel'
    firewall_name:  main
    user_class:     App\Entity\User
    from_email:
        address: "no-reply@joeymasip.com"
        sender_name: "Joey"
    registration:
#        form:
#            type: AppBundle\Form\UserRegisterType
        confirmation:
            enabled: true
            template:   FOSUserBundle:Registration:email.txt.twig
            from_email:
                address:        "no-reply@joeymasip.com"
                sender_name:    "No Reply Registration"
    service:
        mailer: fos_user.mailer.twig_swift
    resetting:
        email:
            template:   FOSUserBundle:Resetting:email.txt.twig

3.2 Creating the User class

namespace App\Entity;

use FOS\UserBundle\Model\User as BaseUser;
use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 * @ORM\Table(name="fos_user")
 */
class User extends BaseUser
{
    /**
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    protected $id;

    public function __construct()
    {
        parent::__construct();
        // your own logic
    }
}

3.3 Configuring main firewall

#config/packages/security.yaml
security:
    encoders:
        FOS\UserBundle\Model\UserInterface: bcrypt
        Symfony\Component\Security\Core\User\User: plaintext

    role_hierarchy:
        ROLE_ADMIN:         ROLE_USER
        ROLE_SUPER_ADMIN:   ROLE_ADMIN

    providers:
        chain_provider:
            chain:
                providers: [in_memory, fos_userbundle]
        in_memory:
            memory:
                users:
                    superadmin:
                        password: 'superadminpw'
                        roles: ['ROLE_SUPER_ADMIN']
        fos_userbundle:
            id: fos_user.user_provider.username

    access_control:
        - { path: ^/login$, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/register, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/resetting, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/admin/, role: ROLE_ADMIN }

    firewalls:
        dev:
            pattern: ^/(_(profiler|wdt)|css|images|js)/
            security: false
        main:
            pattern: ^/
            form_login:
                provider: chain_provider
                csrf_token_generator: security.csrf.token_manager
                login_path: fos_user_security_login
                check_path: fos_user_security_check
                always_use_default_target_path: false
                default_target_path: admin_admin_index

            logout:
                path:   fos_user_security_logout
                target: fos_user_security_login
            anonymous:    true

3.4 Creating the register API endpoint

Now that we have a User entity mapped in our database, let’s create a register API endpoint so we can add new users.

I’ve created an Api folder, and added in routes.yaml

#config/routes.yaml
api:
    prefix: /api
    resource: '../src/Controller/Api'

So all our API endpoints will have the prefix api

So now let’s create a Controller for registering our users.

namespace App\Controller\Api;

use FOS\UserBundle\Model\UserManagerInterface;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use App\Entity\User;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Component\Validator\Validation;
use Symfony\Component\Validator\Constraints as Assert;

/**
 * @Route("/auth")
 */
class ApiAuthController extends AbstractController
{
    /**
     * @Route("/register", name="api_auth_register",  methods={"POST"})
     * @param Request $request
     * @param UserManagerInterface $userManager
     * @return JsonResponse|\Symfony\Component\HttpFoundation\RedirectResponse
     */
    public function register(Request $request, UserManagerInterface $userManager)
    {
        $data = json_decode(
            $request->getContent(),
            true
        );

        $validator = Validation::createValidator();

        $constraint = new Assert\Collection(array(
            // the keys correspond to the keys in the input array
            'username' => new Assert\Length(array('min' => 1)),
            'password' => new Assert\Length(array('min' => 1)),
            'email' => new Assert\Email(),
        ));

        $violations = $validator->validate($data, $constraint);

        if ($violations->count() > 0) {
            return new JsonResponse(["error" => (string)$violations], 500);
        }

        $username = $data['username'];
        $password = $data['password'];
        $email = $data['email'];

        $user = new User();

        $user
            ->setUsername($username)
            ->setPlainPassword($password)
            ->setEmail($email)
            ->setEnabled(true)
            ->setRoles(['ROLE_USER'])
            ->setSuperAdmin(false)
        ;

        try {
            $userManager->updateUser($user, true);
        } catch (\Exception $e) {
            return new JsonResponse(["error" => $e->getMessage()], 500);
        }

        return new JsonResponse(["success" => $user->getUsername(). " has been registered!"], 200);
    }
}
Note: validation and data handling for user creation should be decoupled from the controller, it has been put together just for the example.

If you now send a POST request with the data to http://localhost:8000/api/auth/register, you should get a registered user, and validation error if the data in the json is incorrect or the keys some keys are missing. Also, you’ll get a doctrine error if the username or emails you try to add in the database already exist, as they are unique keys in the FOSUserBundle base User we’re using.

curl -X POST -H "Content-Type: application/json" http://localhost:8000/api/auth/register -d '{"username":"patata","password":"fregida", "email":"patatafregida@joeymasip.com"}'

4. LexikJWTAuthenticationBundle

Now it’s time for the login and recieving a token. For this, we’ll use JWT.

#inside php-fpm bash
composer require lexik/jwt-authentication-bundle

4.1 Private and Public keys

First, let’s create the private and public keys for our project, with a passphrase.

#inside php-fpm bash
mkdir config/jwt
openssl genrsa -out config/jwt/private.pem -aes256 4096
openssl rsa -pubout -in config/jwt/private.pem -out config/jwt/public.pem

4.2 Configuration

Once you’ve created the keys, you can add the config in the yaml and .env files. The passphrase you created the keys with must relate to the config.

#config/packages/lexik_jwt_authentication.yaml
lexik_jwt_authentication:
    secret_key: '%env(resolve:JWT_SECRET_KEY)%'
    public_key: '%env(resolve:JWT_PUBLIC_KEY)%'
    pass_phrase: '%env(JWT_PASSPHRASE)%'
    token_ttl: 3600
#.env
JWT_SECRET_KEY=%kernel.project_dir%/config/jwt/private.pem
JWT_PUBLIC_KEY=%kernel.project_dir%/config/jwt/public.pem
JWT_PASSPHRASE=sf4jwt

4.3 Routes

Once we’ve created the keys and configured the bundle, it’s time to add the login route in our routes.yaml

#config/routes.yaml
api_auth_login:
    path: /api/auth/login
    methods:  [POST]

api:
    prefix: /api
    resource: '../src/Controller/Api'
Note: it's important to put the specific routes before the main ones. See that /api/auth/login is more specific than /api

4.4 Firewalls

And now we have to tell our app to handle this route through configuration, since we won’t be implementing it in our controller.

#config/packages/security.yaml
security:
	#...
    firewalls:
        dev:
            #...

        api_login:
            pattern:  ^/api/auth/login
            stateless: true
            anonymous: true
            json_login:
                provider: chain_provider
                check_path:               /api/auth/login
                success_handler:          lexik_jwt_authentication.handler.authentication_success
                failure_handler:          lexik_jwt_authentication.handler.authentication_failure
            provider: chain_provider

        main:
        	#...

Again, make sure to put api firewalls before the main. The pattern used for the “main” firewall catches everything, the pattern for “api” catches “/api”, so you should put the wildcard AKA main at the end, after the specific cases.

If we now send a POST request to http://localhost:8000/api/auth/login with the username and password from the user we created earlier, you should get a response with the token!

curl -X POST -H "Content-Type: application/json" http://localhost:8000/api/auth/login -d '{"username":"patata","password":"fregida"}'

We’ll get the 200 response

{"token":"eyJ0eXAiOiJKV1QiLCJhbGciOi..."}

All of this is great!

Now it’s time to protect our API calls now that we have tokens right? This is done by creating a new firewall in our security.yaml.

#config/packages/security.yaml
security:
	#...
    firewalls:
        dev:
            #...

        api_login:
            #...

		api:
            pattern: ^/api
            stateless: true
            anonymous: false
            provider: chain_provider
            guard:
                authenticators:
                    - lexik_jwt_authentication.jwt_token_authenticator
        main:
        	#...

Now if we try to call to our previous route http://localhost:8000/api/auth/register without any Authorization header, we’ll see that we get a 401 error.

{
    "code": 401,
    "message": "JWT Token not found"
}

This doesn’t make sense because someone who has to register doesn’t have a token yet! Let’s add one last firewall so anonymous users can register.

#config/packages/security.yaml
security:
	#...
    firewalls:
        dev:
            #...

        api_login:
            #...

        api_register:
            pattern:  ^/api/auth/register
            stateless: true
            anonymous: true

		api:
            #...

        main:
        	#...

To authenticate in our api calls, we just need to add an Authorization header with the Bearer prefix followed by the JWT token, so the value of the header could be for example:

Bearer eyJ0eXAiOiJKV1QiLCJhbGc...

Since we added the anonymous in the /api/auth/register pattern in our firewall, we should now be able to register without sending any token in the header.

Moreover, if you want to open API calls even without a JWT token in the Authorization header, you can just set anonymour to true in the api firewall, like so:

#config/packages/security.yaml
security:
	#...
    firewalls:
        dev:
            #...

        api_login:
            #...

        api_register:
        	#...

		api:
            #...
            anonymous: true
            #...

        main:
        	#...

Now we can add the ACL to control access to fully secure all the prefix routes in our security.yaml

#config/packages/security.yaml
security:
	#...
    access_control:
        #...
        - { path: ^/api/auth/login, roles: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/api/auth/register, roles: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/api, roles: IS_AUTHENTICATED_FULLY }

One last thing, once we register, it would be nice to receive the token inmediately. It would be wierd for an app to make you register and then make you login afterwards. We can redirect to the /auth/login route once a User has been created and return the response.

So in our previous ApiAuthController,

namespace App\Controller\Api;

/**
 * @Route("/auth")
 */
class ApiAuthController extends AbstractController
{
    /**
     * @Route("/register", name="api_auth_register",  methods={"POST"})
     * @param Request $request
     * @param UserManagerInterface $userManager
     * @return JsonResponse|\Symfony\Component\HttpFoundation\RedirectResponse
     */
    public function register(Request $request, UserManagerInterface $userManager)
    {
        #...

        # Code 307 preserves the request method, while redirectToRoute() is a shortcut method.
        return $this->redirectToRoute('api_auth_login', [
            'username' => $data['username'],
            'password' => $data['password']
        ], 307);
    }
}

5. NelmioApiDocBundle

Let’s add some docs in our API project.

#inside php-fpm bash
composer require nelmio/api-doc-bundle

5.1 Configuration

#config/packages/nelmio_api_doc.yaml
nelmio_api_doc:
    documentation:
#        schemes: [http, https]
        info:
            title: Symfony JWT API
            description: Symfony JWT API docs
            version: 1.0.0
        securityDefinitions:
            Bearer:
                type: apiKey
                description: 'Authorization: Bearer {jwt}'
                name: Authorization
                in: header
        security:
            - Bearer: []
    areas: # to filter documented areas
        path_patterns:
            - ^/api(?!/doc$) # Accepts routes under /api except /api/doc

5.2 Routing

Also uncomment the app.swagger_ui route in

#config/routes/nelmio_api_doc.yaml
app.swagger_ui:
    path: /api/doc
    methods: GET
    defaults: { _controller: nelmio_api_doc.controller.swagger_ui }

5.3 ACL

Finally, we want to add this to our ACL

#config/packages/security.yaml
security:
	#...
    access_control:
        #...
        - { path: ^/api/doc, roles: IS_AUTHENTICATED_ANONYMOUSLY }
        #...
        - { path: ^/api, roles: IS_AUTHENTICATED_FULLY }

Again, remember to put this specific route before the generic /api prefix

If we open the browser http://localhost:8000/api/doc we should see the swagger with the jwt api key auth for secured routes.

6. NelmioCorsBundle

Cross-origin resource sharing is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served. So if we want to access our API from a different domain we’ll probably run into CORS problems, so let’s quickly set up NelmioCorsBundle!

#inside php-fpm bash
composer req cors

Change the default configuration from this

#config/packages/nelmio_cors.yaml
nelmio_cors:
    defaults:
        origin_regex: true
        allow_origin: ['%env(CORS_ALLOW_ORIGIN)%']
        allow_methods: ['GET', 'OPTIONS', 'POST', 'PUT', 'PATCH', 'DELETE']
        allow_headers: ['Content-Type', 'Authorization']
        expose_headers: ['Link']
        max_age: 3600
    paths:
        '^/': ~

To this

#config/packages/nelmio_cors.yaml
nelmio_cors:
    defaults:
        allow_credentials: false
        allow_origin: []
        allow_headers: []
        allow_methods: []
        expose_headers: []
        max_age: 0
        hosts: []
        origin_regex: false
        forced_allow_origin_value: ~
    paths:
        '^/api/':
            allow_origin: ['*']
            allow_headers: ['Content-Type', 'Authorization']
            allow_methods: ['POST', 'PUT', 'GET', 'DELETE']
            max_age: 3600
        '^/':
            origin_regex: true
            allow_origin: ['^http://localhost:[0-9]+']
            allow_headers: ['Content-Type', 'Authorization']
            allow_methods: ['POST', 'PUT', 'GET', 'DELETE']
            max_age: 3600
            hosts: ['^api\.']

This will allow all origins for the /api prefix so any mobile app can now use our API. Feel free to play with the regex until you’re comfortable with the result.

7. Creating an example API enpoint

As an example, let’s create an API endpoint for retrieving a user.

I’ll create a new controller

namespace App\Controller\Api;

use App\Entity\User;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Component\Serializer\Serializer;
use Symfony\Component\Serializer\Encoder\XmlEncoder;
use Symfony\Component\Serializer\Encoder\JsonEncoder;
use Symfony\Component\Serializer\Normalizer\ObjectNormalizer;

/**
 * @Route("/user")
 */
class ApiUserController extends AbstractController
{
    /**
     * @Route("/{id}", name="api_user_detail", methods={"GET"})
     * @param User $user
     * @return JsonResponse
     */
    public function detail(User $user)
    {
        $this->denyAccessUnlessGranted('view', $user);
        return new JsonResponse($this->serialize($user), 200);
    }

    protected function serialize(User $user)
    {
        $encoders = [new XmlEncoder(), new JsonEncoder()];
        $normalizers = [new ObjectNormalizer()];

        $serializer = new Serializer($normalizers, $encoders);

        $json = $serializer->serialize($user, 'json');

        return $json;
    }
}
Note: Please consider serializing in a separate service, this is a just some example code.

We’ll need a voter to let us know if a user is allowed to be retrieved or not. It should only be retrieved if it’s himself, or if it’s an admin right?

namespace App\Security;

use App\Entity\User;
use Symfony\Component\Security\Core\Authentication\Token\TokenInterface;
use Symfony\Component\Security\Core\Authorization\Voter\Voter;
use Symfony\Component\Security\Core\Authorization\AccessDecisionManagerInterface;

class UserVoter extends Voter
{
    // these strings are just invented: you can use anything
    const VIEW = 'view';
    const EDIT = 'edit';

    private $decisionManager;

    public function __construct(AccessDecisionManagerInterface $decisionManager)
    {
        $this->decisionManager = $decisionManager;
    }

    protected function supports($attribute, $subject)
    {
        // if the attribute isn't one we support, return false
        if (!in_array($attribute, array(self::VIEW, self::EDIT))) {
            return false;
        }

        // only vote on User objects inside this voter
        if (!$subject instanceof User) {
            return false;
        }

        return true;
    }

    protected function voteOnAttribute($attribute, $subject, TokenInterface $token)
    {
        $user = $token->getUser();

        if (!$user instanceof User) {
            // the user must be logged in; if not, deny access
            return false;
        }

        // ROLE_SUPER_ADMIN can do anything! The power!
        if ($this->decisionManager->decide($token, array('ROLE_ADMIN'))) {
            return true;
        }

        // you know $subject is a User object, thanks to supports
        /** @var User $userSubject */
        $userSubject = $subject;

        switch ($attribute) {
            case self::VIEW:
                return $this->canView($userSubject, $user);
            case self::EDIT:
                return $this->canEdit($userSubject, $user);
        }

        throw new \LogicException('This code should not be reached!');
    }

    private function canView(User $userSubject, User $user)
    {
        // if they can edit, they can view
        if ($this->canEdit($userSubject, $user)) {
            return true;
        }

        // the User object could have, for example, a method isPrivate()
        // that checks a boolean $private property
        return $user === $userSubject;
    }

    private function canEdit(User $userSubject, User $user)
    {
        // this assumes that the data object has a getOwner() method
        // to get the entity of the user who owns this data object
        return $user === $userSubject;
    }
}

As you can see, we can get the user from the TokenInterface. The function

$tokenInterface->getUser()

will return a UserInterface aka our User. The TokenInterface can be injected in an EventListener, Controller, etc. so you can always know what user is making the request.

More on how to use voters for securing your app here.

Let’s try getting that user with the GET request.

curl -X GET -H "Content-Type: application/json" http://localhost:8000/api/user/1

The response is a 401

{"code":401,"message":"JWT Token not found"}

This time let’s add the token in the header, like so

curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE1NDgyODI4NjUsImV4cCI6MTU0ODI4NjQ2NSwicm9sZXMiOlsiUk9MRV9VU0VSIl0sInVzZXJuYW1lIjoicGF0YXRhIn0.SLB40jBqtbY7Ql5DI4L4rZBEcH5hXNqn9u0eVYOCtlE_vqa_1RYQzHHVy7iMbmP4CMjSKUiBlGzuIBGApRmD36CFKgdMdfXuqrHeEFB-BUsd-HezdLg6U762GnLbe4g4vHzg9XKZVCzRtpboGUKgVycIaMdfiZ1FvUkJeZvdS_5HgHW43LrSqntPlbEaNaEYv7mrkzTQNi1WiKwJaDCW5M0JgRkfbHoUYXMadFUxR4KSnaXRQAwxZnqqPBW4dRs97ho5A15XKpuxmEWemvhMs0XL9E8KyPNG7ZipLis4JGs3X9Mn-ov4RDCqYbShSNbtj_F2gcakXL97FF3myLn1U2XfIzuwxq9ZIJnGLtemKgPlxSB1uxX5ep9aYxppuXpwxY0vGr9MsOgyL3kkuMqeXvFDN46bY-3P8TLOqEuPrrlKYuRfMQv6Wrhdq0orl3eo7t83YCb_Z-Mf7yeDDGeJsftaj4pALJUw4Ovo6Kv_4gNcG3VQpkJr4XtnULAcO9O_OJLgVOBXoOc7lUQmokdvAGeltEBmYIZD_2KtGrTwS4rL55LMn3MawL4dKVIAg8aaYbPDCxkk1t3LdZyI5zSUiJvLaCrMM8ZhJ7eJ0rKod2-d_dZcYPzQ5RF_wD8spuw1pkT6r4hMyTJvGQUUDjN-3E-MkNBHT8Ku8Z8I7a65x5M" http://localhost:8000/api/user/1

This will return the user serialized, so success!

Conclusion

We’ve created a project from scratch and we’ve installed a User library, a JWT library, an API doc library and a CORS library. We’ve created a register endpoint, we’ve protected our api with firewalls and we’ve created a user endpoint with a voter to test authentication and authorization.

This is a great start to develop your API project and scale it up!

Happy coding! 🙂

Resources

Link to symfony4-api-jwt GitHub project.
Link to all commits.

Docker to set up the development environment.
LexikJWTAuthenticationBundle to authenticate users on our API side.
FOSUserBundle for User entities.
NelmioApiDocBundle for our API docs.
NelmioCorsBundle for Cross Origin.

Django project, apps structure and folders

In this blog post I’ll talk about Django folders structure inside a project.

After developing a few projects with Django 1.11 and Django 2.0, I’ve stumbled with somewhat an issue that’s been bothering me. When you create projects and apps in Django, as the tutorial shows you, apps will be created inside the main project folder, at the same level of the project’s settings folder (usually it’s the folder that has the same name as your project).

So basically, what you end up with is a container folder and inside has the project config folder, and many other folders with the app names.

So as a result, you get different ordering in the folders for every new project, as folders will usually order by name inside your IDE, i.e PyCharm.

So, for instance, if your app is called ‘apples’ and your project is called ‘mysite’, then the order will be:

- apples
- mysite

Let’s also assume you’re using docker, so you probably have a docker folder, now we have:

- apples
- docker
- mysite

But if you create an app called ‘oranges’, now your project folder is:

- apples
- docker
- mysite
- oranges

This is unconfortable because you never know which is the project’s config folder and which ones are the different app folders, so you keep opening the wrong folders all the time!! This is awful, as programers, we tend strive for efficiency!! 🙂

So my strategy for now is to bundle all apps inside an apps folder, so my project will always look like:

- apps
- docker
- mysite

And there you have it, nicely ordered apps inside your project!!

To include them in your settings, you just have to remember to include the namespace

'apps.your_app_name'

so if your app is called apples

INSTALLED_APPS += ['apps.apples']

Also, when importing classes in different files, just use the namespace.

So imagine we need to import the Apple class defined in my models inside my apples app, would do something like so:

from apps.apples.models import Apple

If you’ve read until here, it means you REALLY CARE about your project folder structure, so here’s a handy script I’ve created to start apps. It’s just a wrapper from the django startapp command, but it takes care of the apps folder, etc.

#!/usr/bin/env bash

if [ "$1" == "-h" ]; then
	echo "This script will create an app inside the apps folder"
	echo "To use type the following line:"
	echo "bash start-app.sh app_name"
	echo "Replace app_name with the actual name for your app"
elif [ "$1" != "" ]; then
    if [ ! -d "apps" ]; then
        mkdir apps
        touch apps/__init__.py
    fi
    mkdir apps/$1
    if [ -f /.dockerenv ]; then
        python manage.py startapp $1 apps/$1
    else
        docker-compose run django python manage.py startapp $1 apps/$1
    fi
    echo "Success! The app $1 has been aded, don't forget to add INSTALLED_APPS += ['apps.$1'] in your project's settings.py"
else
	echo "Error! One parameter is expected: app_name"
fi

I call the file “start-app.sh”, so to use it just type

bash start-app.sh app_name

Happy coding!!

Gedmo Translations in Symfony 4

Today I’ve had to install Gedmo Translations into a Symfony 4 app and I had some trouble, so after fixing them problems I thought of writing them down in case someone else can benefit from it 🙂

Let’s start fromt he begginig.

Create a Symfony 4 project

Please use docker! 🙂 More info here in the Docker for Symfony 4 post.

Here’s what my docker.compose.yml looks like

#docker.compose.yml
version: "3.1"

volumes:
    db-data:

services:
    mysql:
      image: mysql:5.6
      container_name: ${PROJECT_NAME}-mysql
      working_dir: /application
      volumes:
        - db-data:/application
      environment:
        - MYSQL_ROOT_PASSWORD=docker_root
        - MYSQL_DATABASE=gedmoapp_db
        - MYSQL_USER=gedmoapp_user
        - MYSQL_PASSWORD=gedmoapp_pw
      ports:
        - "8306:3306"

    webserver:
      image: nginx:alpine
      container_name: ${PROJECT_NAME}-webserver
      working_dir: /application
      volumes:
        - .:/application
        - ./docker/nginx/nginx.conf:/etc/nginx/conf.d/default.conf
      ports:
        - "8000:80"

    php-fpm:
      build: docker/php-fpm
      container_name: ${PROJECT_NAME}-php-fpm
      working_dir: /application
      volumes:
        - .:/application
        - ./docker/php-fpm/php-ini-overrides.ini:/etc/php/7.2/fpm/conf.d/99-overrides.ini

Installation of Gedmo’s bundle

composer require stof/doctrine-extensions-bundle

Configuration

Now we need to update our configuration file, so in the doctrine.yaml, under the doctrine key, we should go from this

#doctrine.yaml
doctrine:
#...
    orm:
        auto_generate_proxy_classes: '%kernel.debug%'
        naming_strategy: doctrine.orm.naming_strategy.underscore
        auto_mapping: true
        naming_strategy: doctrine.orm.naming_strategy.underscore
        auto_mapping: true
        mappings:
            App:
                is_bundle: false
                type: annotation
                dir: '%kernel.project_dir%/src/Entity'
                prefix: 'App\Entity'
                alias: App

To this

#doctrine.yaml
doctrine:
#...
	orm:
        auto_generate_proxy_classes: '%kernel.debug%'
#        naming_strategy: doctrine.orm.naming_strategy.underscore
#        auto_mapping: true
        entity_managers:
            default:
#                connection: default
                naming_strategy: doctrine.orm.naming_strategy.underscore
                auto_mapping: true
                mappings:
                    App:
                        is_bundle: false
                        type: annotation
                        dir: '%kernel.project_dir%/src/Entity'
                        prefix: 'App\Entity'
                        alias: App
                    gedmo_translatable:
                        type: annotation
                        prefix: Gedmo\Translatable\Entity
                        dir: "%kernel.root_dir%/../vendor/gedmo/doctrine-extensions/lib/Gedmo/Translatable/Entity"
                        alias: GedmoTranslatable # (optional) it will default to the name set for the mapping
                        is_bundle: false
                    gedmo_translator:
                        type: annotation
                        prefix: Gedmo\Translator\Entity
                        dir: "%kernel.root_dir%/../vendor/gedmo/doctrine-extensions/lib/Gedmo/Translator/Entity"
                        alias: GedmoTranslator # (optional) it will default to the name set for the mapping
                        is_bundle: false
#                    gedmo_loggable:
#                        type: annotation
#                        prefix: Gedmo\Loggable\Entity
#                        dir: "%kernel.root_dir%/../vendor/gedmo/doctrine-extensions/lib/Gedmo/Loggable/Entity"
#                        alias: GedmoLoggable # (optional) it will default to the name set for the mappingmapping
#                        is_bundle: false
                    gedmo_tree:
                        type: annotation
                        prefix: Gedmo\Tree\Entity
                        dir: "%kernel.root_dir%/../vendor/gedmo/doctrine-extensions/lib/Gedmo/Tree/Entity"
                        alias: GedmoTree # (optional) it will default to the name set for the mapping
                        is_bundle: false

Also, in our stof_doctrine_extensions.yaml, let’s add the following configuration for the doctrine extensions.

#stof_doctrine_extensions.yaml
stof_doctrine_extensions:
    default_locale: en_US
    translation_fallback: true
    persist_default_translation: true
    orm:
        default:
            tree: true
            translatable: true
            sluggable: true

Now, if we try to update our schema

php bin/console doc:sch:update --dump-sql
The following SQL statements will be executed:

     CREATE TABLE ext_translations (id INT AUTO_INCREMENT NOT NULL, locale VARCHAR(8) NOT NULL, object_class VARCHAR(255) NOT NULL, field VARCHAR(32) NOT NULL, foreign_key VARCHAR(64) NOT NULL, content LONGTEXT DEFAULT NULL, INDEX translations_lookup_idx (locale, object_class, foreign_key), UNIQUE INDEX lookup_unique_idx (locale, object_class, field, foreign_key), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci ENGINE = InnoDB ROW_FORMAT = DYNAMIC;

Looks about right, so let’s hit it!

php bin/console doc:sch:update --force

And we get the following error

Updating database schema...


In AbstractMySQLDriver.php line 126:
                                                                                                                                
  An exception occurred while executing 'ALTER TABLE ext_translations CHANGE object_class object_class VARCHAR(255) NOT NULL':  
                                                                                                                                
  SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes               
                                                                                                                                

In PDOConnection.php line 109:
                                                                                                                   
  SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes  
                                                                                                                   

In PDOConnection.php line 107:
                                                                                                                   
  SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes

Now, after looking for this issue with Symfony and Gedmo Translations in the internet, I found this issue about Doctrine and this other one, and the fix here.

As davidbehler and javiereguiluz explain in the posts,

the cause of this is that Symfony advises you to use utf8mb4_general_ci/utf8mb4 as collation/charset for your database. utf8mb4 takes 4 bytes per char, meaning a 255 char field needs 1020 bytes for an index.

And so it seems that MySql 5.6 has a max key length of 767 bytes, so that leaves us with three options:

1. Decrease the field length to 191. However, we would have to override Gedmo’s translations bundle, as it uses 255 char fields everywhere…

2. We could change the charset encoding to ut8 like so (thanks to Kuba Florczuk for the configuration):

#doctrine.yaml
doctrine:
    dbal:
        # configure these for your database server
        driver: 'pdo_mysql'
        server_version: '5.6'
        charset: utf8
        default_table_options:
            charset: utf8
            collate: utf8_unicode_ci

        url: '%env(resolve:DATABASE_URL)%'
#...

3. We could upgrade to mysql 5.7 like so:

#docker-compose.yml
services:
    mysql:
      image: mysql:5.7
#...
#doctrine.yaml
doctrine:
    dbal:
        # configure these for your database server
        driver: 'pdo_mysql'
        server_version: '5.7'
        charset: utf8mb4
        default_table_options:
            charset: utf8mb4
            collate: utf8mb4_unicode_ci

        url: '%env(resolve:DATABASE_URL)%'
#...

All of them solutions work gracefully. Hope this helps, happy coding! 🙂

Gedmo’s Translations Documentation

StofDoctrineExensionBundle in Symfony’s official website.
Doctrine’s extensions GitHub doc.
Translatable’s GitHub doc.

How to build a REST API with Django and JWT Auth

It’s been a while since I wrote anything in the blog.

Today on this short tutorial I’ll explain the steps on how to build an api app with Django, using the JWT (JSON Web Token) as a way to identify users.

Libraries we’ll use are:

Docker
Django 1.11
Python 3
Django’s REST Framework
Django’s JWT

So let’s start by creating a Django project with Docker. If you still don’t know how to do that, more info on how to do create a django project with docker here.

Our requirements.txt would something like this:

django==1.11
mysqlclient
djangorestframework
djangorestframework-jwt
requests

*Note: The requests library is just to create some requests (GET, POST…) inside our python code.

For this example, I’ve named the project ‘apiskeleton’ and the app ‘apiapp’.

Step1: Django’s REST Framework

Once we have created the project and app, let’s register our app and the rest framework app also in our settings.

#settings.py

INSTALLED_APPS = [
	# ...
	'apiapp',
    'rest_framework',
	# ...
]

Great, now we’re ready to create some logic in our app.

First off, if you haven’t done already, create a separate User model. This is best practices for Django.

#models/user.py
from django.db import models
from django.contrib.auth.models import AbstractUser


class User(AbstractUser):
    pass

Update the settings.py

#settings.py

# Custom User model
AUTH_USER_MODEL = 'apiapp.User'

Now, let’s create a serializer for our User model. The serializer will allow us to convert a model instance or queryset to a JSON content type. More info on serializers here.

Our serializers.py would look something like this:

#serializers/user.py
from rest_framework import serializers
from appapi.models import User

class UserSerializer(serializers.Serializer):
    id = serializers.IntegerField(read_only=True)
    username = serializers.CharField(required=True, allow_blank=False, max_length=100)
    password = serializers.CharField(required=True, write_only=True)
    email = serializers.CharField(required=True, allow_blank=True, max_length=100)
    is_staff = serializers.BooleanField(required=False, default=False)
    is_superuser = serializers.BooleanField(required=False, default=False)

    def create(self, data):
        """
        Create and return a new `Snippet` instance, given the validated data.
        """
        instance = User.objects.create(
            username=data.get('username'),
            email=data.get('email'),
            is_staff=data.get('is_staff'),
            is_superuser=data.get('is_superuser'),
        )
        instance.set_password(data.get('password'))
        instance.save()
        return instance

    def update(self, instance, data):
        """
        Update and return an existing `Snippet` instance, given the validated data.
        """
        instance.username = data.get('username', instance.username)
        instance.email = data.get('email', instance.email)
        instance.is_staff = data.get('is_staff', instance.is_staff)
        instance.is_superuser = data.get('is_superuser', instance.is_staff)
        instance.set_password(data.get('password'))
        instance.save()
        return instance 

It’s very simple, I know, but I like the readability here! There are probably better ways to make your serializers, for instance, by extending serializers.ModelSerializer, more info on this here, but it’s not the aim of this tutorial right now. Trying to make things as easy and readable as possible.

Now let’s add a GET API route to list all users, something on the lines of /user/. Since it’s listing all users, it should only be allowed for an admin user to use it right? For now, I’ll just call it api_admin_user_index, so I know this is for admins only. Later on we’ll secure these.

#urls.py
urlpatterns = [
    url(r'^user/$', user.api_admin_user_index, name='api_admin_user_index'),
]

Now, let’s create the view for this url:

#views/user.py
from rest_framework.decorators import api_view
from rest_framework.response import Response
from apiapp.models import User
from apiapp.serializers.user import UserSerializer

@api_view(['GET'])
def api_admin_user_index(request):
    """
    get:
    List all users.
    """
    serializer = UserSerializer(User.objects.all(), many=True)
    return Response(serializer.data)

This first parameter in the serializer is the queryset, and the second parameter is to let the serializer know this is a list. More info on this on the BaseSerializer class from Django’s Rest Framework.

Now, if we open a browser and type this route, we should see a list of 0 users, but the route works right? Success!

Now let’s create some users. We should add the POST verb in our view to do that, so it would look something like this:

#views/user.py
from rest_framework import status
from rest_framework.decorators import api_view
from rest_framework.response import Response
from apiapp.models import User
from apiapp.serializers.user import UserSerializer
from apiapp.utils.jsonreader import JsonReader


@api_view(['GET', 'POST'])
def api_admin_user_index(request):
    """
    get:
    List all users.
    post:
    Create new user.
    """
    if request.method == 'GET':
        serializer = UserSerializer(User.objects.all(), many=True)
        return Response(serializer.data)

    elif request.method == 'POST':
        data = JsonReader.read_body(request)
        serializer = UserSerializer(data=data)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data, status=status.HTTP_201_CREATED)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

The POST will create a user through the serializer.

In the code, you can see that I added a JsonReader class. This is a util class I use in most of my projects, it just makes reading the body easier, here’s the code:

#utils/jsonreader.py
import json


class JsonReader:

    @staticmethod
    def read_body(request):
        '''returns a dict from the body of request'''

        data = dict()

        try:
            body = request.body
            data = JsonReader.bytes_to_dict(body)
        except Exception as exc:
            if isinstance(request.data, dict):
                data = request.data
            elif isinstance(request.data, str):
                data = JsonReader.str_to_dict(request.data)

        return data

    @staticmethod
    def str_to_dict(s: str):
        dict = json.loads(s)
        return dict

    @staticmethod
    def dict_to_str(d: dict):
        str = json.dumps(d)
        return str

    @staticmethod
    def bytes_to_dict(b: bytes):
        str = b.decode("utf-8")
        dict = JsonReader.str_to_dict(str)
        return dict

Now we can create some users through our POST method, just sending some JSON with the keys username, password and email, which are the required ones in our serializer. Simple right?

Now it’s time for the rest of the usual verbs, PUT and DELETE. Also, we need a GET for a single user.

The routes would look something like this:

#urls.py
urlpatterns = [
    url(r'^user/$', user.api_admin_user_index, name='api_admin_user_index'),
    url(r'^user/(?P[0-9]+)/$', user.api_admin_user_detail, name='api_admin_user_detail'),
]

And now let’s update the view

#views/user.py
@api_view(['GET', 'PUT', 'DELETE'])
def api_admin_user_detail(request, pk):
    """
    get:
    Detail one user.
    put:
    Update one user.
    delete:
    Delete one user.
    """

    try:
        user = User.objects.get(pk=pk)
    except User.DoesNotExist:
        return Response({'error': "User " + pk + " does not exist"}, status=status.HTTP_404_NOT_FOUND)

    if request.method == 'GET':
        serializer = UserSerializer(user)
        return Response(serializer.data)

    elif request.method == 'PUT':
        data = JsonReader.read_body(request)
        serializer = UserSerializer(user, data=data)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

    elif request.method == 'DELETE':
        user.delete()
        return Response(status=status.HTTP_204_NO_CONTENT)

All of this is great, but now let’s secure these admin calls right? Here’s where JWT comes into play.

Step2: Django’s JWT

JWT is a stateless authentication mechanism as the user state is never saved in the server memory. Django will check for a valid JWT in the Authorization header, and if it is there, the user will be allowed. As JWTs are self-contained, all the necessary information is there, reducing the need to go back and forth to the database.

This allows the user to fully rely on data APIs that are stateless. It doesn’t matter which domains are serving your APIs, as Cross-Origin Resource Sharing (CORS) won’t be an issue since it doesn’t use cookies.

With that being said, let’s start by updating our settings.py with the JWT configuration

#settings.py

# Configure the authentication in Django Rest Framework to be JWT
# http://www.django-rest-framework.org/
REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework_jwt.authentication.JSONWebTokenAuthentication',
    ),
}

# Configure the JWTs to expire after 1 hour, and allow users to refresh near-expiration tokens
# https://getblimp.github.io/django-rest-framework-jwt/
JWT_AUTH = {
    # If the secret is wrong, it will raise a jwt.DecodeError telling you as such. You can still get at the payload by setting the JWT_VERIFY to False.
    'JWT_VERIFY': True,

    # You can turn off expiration time verification by setting JWT_VERIFY_EXPIRATION to False.
    # If set to False, JWTs will last forever meaning a leaked token could be used by an attacker indefinitely.
    'JWT_VERIFY_EXPIRATION': True,

    # This is an instance of Python's datetime.timedelta. This will be added to datetime.utcnow() to set the expiration time.
    # Default is datetime.timedelta(seconds=300)(5 minutes).
    'JWT_EXPIRATION_DELTA': datetime.timedelta(hours=1),

    'JWT_ALLOW_REFRESH': True,
    'JWT_AUTH_HEADER_PREFIX': 'JWT',
}

The config is pretty self-explanatory, but there’s more documentation on JWT settings here.

Once this is out of the way, let’s add some JWT route patterns so we can get that JWT token.

#urls.py
urlpatterns = [
    # ...
    url(r'^jwt/refresh-token/', refresh_jwt_token, name='refresh_jwt_token'),
    url(r'^jwt/api-token-verify/', verify_jwt_token, name='verify_jwt_token'),
    url(r'^jwt/api-token-auth/', obtain_jwt_token, name='obtain_jwt_token'),
]

Also, this is not necessary, but instead of exposing the JWT routes, let’s add a login API route that will wrap the obtaining of the JWT token.

#urls.py
urlpatterns = [
    # ...
    url(r'^login/', registration.api_login, name='api_login'),
]

And in our registration view

#views/registration.py
@api_view(['POST'])
def api_login(request):
    """
    post:
    This view is called through API POST with a json body like so:

    {
        "username": "admin",
        "password": "admin"
    }

    :param request:
    :return:
    """
    data = JsonReader.read_body(request)

    response_login = requests.post(
        request.build_absolute_uri(reverse('obtain_jwt_token')),
        data=data
    )
    response_login_dict = json.loads(response_login.content)
    return Response(response_login_dict, response_login.status_code)

Notice that this time I didn’t use the prefix ‘admin’. This is because this will be authorized to anonymous users, unlike the ones we created previously.

So as you can see, it’s pretty straightforward. This view is a wrapper of the JWT route. This way, we can add information to the response if we need to, for instance, the user id, user profile, etc. For now, let’s leave it this way.

If we now test it with a username and password, we should get the token. If you haven’t created any users yet, you could use the POST call we created previously. Otherwise, just add some superusers in the python console or create some fixtures, up to you.

If you’re lazy, here are my fixtures, and the location is

apiapp/fixtures/users.json
[
{
    "model": "apiapp.user",
    "pk": 1,
    "fields": {
        "password": "pbkdf2_sha256$36000$SoF7ueJzOE1I$244qaRI1dReT4DxZKXJi2sRmuKdqPcjeOiUaPdH2UV0=",
        "last_login": null,
        "is_superuser": true,
        "username": "admin",
        "first_name": "",
        "last_name": "",
        "email": "admin@slowcode.io",
        "is_staff": true,
        "is_active": true,
        "date_joined": "2018-08-29T10:09:27.191Z",
        "groups": [],
        "user_permissions": []
    }
},
{
    "model": "apiapp.user",
    "pk": 2,
    "fields": {
        "password": "pbkdf2_sha256$36000$bkA3NXYqXZ4S$zuH97poSj3trZoNRFeROw3PgutbGOHlZunliI8/1jbg=",
        "last_login": null,
        "is_superuser": false,
        "username": "user1",
        "first_name": "",
        "last_name": "",
        "email": "user1@slowcode.io",
        "is_staff": true,
        "is_active": true,
        "date_joined": "2018-08-29T10:10:56.873Z",
        "groups": [],
        "user_permissions": []
    }
},
{
    "model": "apiapp.user",
    "pk": 3,
    "fields": {
        "password": "pbkdf2_sha256$36000$KKujn28LvdGX$OxtfRmIUWNPwbBPsz2iwKwff8klJ5PiPXj3P9N70Hto=",
        "last_login": null,
        "is_superuser": false,
        "username": "user2",
        "first_name": "",
        "last_name": "",
        "email": "user2@slowcode.io",
        "is_staff": false,
        "is_active": true,
        "date_joined": "2018-08-29T10:11:19.576Z",
        "groups": [],
        "user_permissions": []
    }
}
]

To import them, use the command

python manage.py loaddata users

Great, by now you should have some users in your database and you should be able to use our login API we just created. But what do we do with the token we get back?

Well, as mentioned before, this token should be added as a Authorization header in our calls to identify such user.

With JWT we will be able to authorize or prohibit certain calls for certain users, and or we will send different type of info, depending on what user is sending the call.

All of this decision making will be done by a different layer, which I call ‘security voters’ (hereditary from Symfony)

Here’s an example of what a voter would look like:

#security/voters.py
from apiapp.models import User


class AbstractVoter:

    request = None

    def __init__(self, request):
        self.request = request

    def is_logged_in(self):
        if isinstance(self.request.user, User):
            return True

        return False

    def is_superuser(self):
        if self.is_logged_in():
            return self.request.user.is_superuser

        return False


class UserVoter(AbstractVoter):

    def user_can_manage_me(self, user_inst: User):
        if self.is_logged_in():
            if self.is_superuser():
                return True
            if self.request.user == user_inst:
                return True

        return False

Now we can call this voter in our APIs.

#views/user.py
@api_view(['GET', 'POST'])
def api_admin_user_index(request):
    """
    get:
    List all users.
    post:
    Create new user.
    """
    voter = UserVoter(request)
    if not voter.is_superuser():
        return Response({'error': "User API is not allowed by non admin user"}, status=status.HTTP_403_FORBIDDEN)

    #...


@api_view(['GET', 'PUT', 'DELETE'])
def api_admin_user_detail(request, pk):
    """
    get:
    Detail one user.
    put:
    Update one user.
    delete:
    Delete one user.
    """
    voter = UserVoter(request)
    if not voter.is_superuser():
        return Response({'error': "User API is not allowed by non admin user"}, status=status.HTTP_403_FORBIDDEN)

    #...

It can also be useful for API calls that are not for superusers. Let’s create the route for a user detail.

#urls.py
urlpatterns = [
	#...
    url(r'^(?P[0-9]+)/$', user.api_user_detail, name='api_user_detail'),
]

And now in the view.
In the GET verb, Unless the user wants to see himself, it should be denied.

#views/user.py
@api_view(['GET', 'PUT'])
def api_user_detail(request, pk):
    """
    get:
    Detail one user.
    put:
    Update one user.
    """
    try:
        user_inst = User.objects.get(pk=pk)
    except User.DoesNotExist:
        return Response(status=status.HTTP_404_NOT_FOUND)

    voter = UserVoter(request)
    if not voter.user_can_manage_me(user_inst):
        return Response({'error': "User API is not allowed"}, status=status.HTTP_403_FORBIDDEN)

    if request.method == 'GET':
        serializer = UserSerializer(user_inst)
        return Response(serializer.data)

    elif request.method == 'PUT':
        data = JsonReader.read_body(request)
        if 'is_staff' in data:
            if not voter.is_superuser():
                return Response({'error': "Non admin cannot update admin attributes"}, status=status.HTTP_403_FORBIDDEN)
        serializer = UserSerializer(user_inst, data=data)
        if serializer.is_valid():
            serializer.save()
            return Response(serializer.data)
        return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

Conclusion

On this post we’ve seen how to implement the Django’s REST Framework together with Django’s JWT library.

We’ve seen that JWT is a powerful and easy way to identify users that use our API.

If you have any questions or something is confusing, let me know and I’ll do my best to clarify.

I’ve also uploaded the code on github so you can clone the project and try it out for yourself.

git clone https://github.com/joeymasip/django-apiskeleton.git

I use it as a base for API projects, as I usually always need some User API to start with.

Happy coding 🙂

Running PHPUnit tests before git push

What usually happens

“Great, I’ve just finished this new feature and it works perfectly, let’s merge it with the rest of the code so the team can now benefit from it..”

git add .
git commit -m "-blablabla"
git flow feature finish "blabla-feature-name"
git push origin develop

“Oops, I forgot to pass tests after closing the feature branch…”

./vendor/bin/simple-phpunit

...F
FAILURES!
Tests:4... Failures: 1

We’ve all done it before, pushed code that didn’t pass tests…

Pre-Push git hook solution

On this blog post I’ll show you how to set a pre-push script, so every time you push code to your remote repository (develop or master only for this example), it’ll try to run your tests. And as a result, it will only push the code if tests pass.

This is intended to be like a safety net, specially if you’re working with a team and your code needs to be tested before being pushed!

This is what the script file looks like:

#!/usr/bin/env php

< ?php

/**
 * ===========================
 * This hook will only allow git push if your all test pass
 * To activate this hook, you need to place this file in the .git/hooks folder
 * ===========================
 **/


$gitBranch = shell_exec('git branch | grep \*');
$branchName = trim(str_replace('*', '', $gitBranch));
if( $branchName !== 'develop' && $branchName !== 'master' ) {
    echo "You're not in master or develop, so hook tests won't activate.." . PHP_EOL;
    exit(0);
}

echo "Running tests.. ";
exec('./vendor/bin/simple-phpunit', $output, $returnCode);
if ($returnCode !== 0) {
  // Show full output
  echo PHP_EOL . implode($output, PHP_EOL) . PHP_EOL;
  echo "Cannot push changes untill tests are OK.." . PHP_EOL;
  exit(1);
}

// Show summary (last line)
echo array_pop($output) . PHP_EOL;
exit(0);
Note1: Notice the start of script tag < ?php . You will need to remove the white space so PHP interprets the code.
Note2: The name of this file must be pre-push and must be located inside the .git/hooks folder in your project. This is because git will look for that actual file before pushing.

So if we dive into the code, it's pretty simple.

The first block looks in what branch you're located. I've just set up this script to run tests when you try and push to develop or master. You can modify/delete this part if you want a different behaviour.

The second block is the actual running of the tests. This only runs if you're located in develop or master.

As you can see, if return code after running tests is different than 0, AKA a tests failed, it'll exit the script with an exit(1), letting git know it must not push the code.

If all test pass, it'll exit with an exit(0) and so, the code will be pushed.

And that's it! Simple right?

There are other git hooks that might be useful for you. Try looking at the .git/hooks folder for samples (pre-commit, pre-rebase, pre-receive...) and let me know if you use any other ones!

Cheers and safe coding! 🙂

Docker for Django

This blog post is a small guide for getting started with your Django environment with Docker. Since I got very positive feedback from the blog post about Docker and Symfony4, I decided to do the same with Docker and Django.

In this example we’re going to work with Django 1.11 (LTS), Python3 and MySql 5.6

Before we start, you’ll need to install Docker in your machine. You can download it from the official website.

Once Docker is installed, I strongly recommend playing with the getting started guide. Here the guide for macs and here the guide for windows. However, if you’re lazy like me, just use this command to make sure it’s installed.

docker --version

Once all of that is out of the way, we can start with the Django project and the Docker environment that will run it.

We have two cases I’m going to tap on.

Case 1: I’m creating a Django project from scratch and I want to set up a development environment with Docker.

Step1: Clone the docker-django repository which has the docker configuration files.

git clone https://github.com/joeymasip/docker-django.git

Step2: Create the Django project.

First off, let’s start docker containters with the project we just downloaded.

#cd to the location where you cloned the project
cd ~/Development/docker-django
#start the containers
docker-compose up -d 

This command starts the containers. The parameter -d makes them run in the background. If you omit the -d you’ll see the log.

Docker should start building (if it’s the first time for these images) and running with the containers in the background.

If you now try to run

docker ps

in your console, you’ll see that MySql container is running, but Django’s is not. This is normal, as we haven’t installed Django yet in our project.

Now we’ll create the django project with the following command (replace project_name for your project name)

docker-compose run django django-admin.py startproject project_name .

Note: Do not forget the . in the end

Step3: Create the Django application.

Now we’ll create the django application with the following command (replace app_name for your app’s name)

docker-compose run django python manage.py startapp app_name

Now, try running the same

docker-compose up -d

Now, if you run the docker ps

docker ps

This time, Django’s container will have been started.

So right now, if you just open your browser and type http://127.0.0.1:8000/ you should see it working, so you’re already set up to develop!

First off though, let’s update settings.py so we can use our mysql container instead of Django’s sqlite.

Step4: Update settings.

First let’s add your app_name in the installed apps

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    
    'app_name',
]

Now let’s configure your settings.py so the database settings points to the database service from docker.

Change:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

To:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'docker_django_db',
        'USER': 'dbuser',
        'PASSWORD': 'dbpw',
        'HOST': 'mysql',
        'PORT': '3306',
        'TEST': {
            'NAME': 'docker_django_db_test',
        },
    }
}

Step5: Create the User model.

Under your app_name/models.py file, just create a User model that extends from Django’s Auth model.

#app_name/models.py
from django.db import models
from django.contrib.auth.models import AbstractUser


class User(AbstractUser):
    pass

Once this is done, just tell your project that we’ll be using our User model. This is best practices in case you ever need to make changes to the User model. So in your project_name settings.py, add this line:

AUTH_USER_MODEL = 'app_name.User'

Now our User is plugged in.

Step6: Run migrations.

To run migrations, we need to first enter the python django bash.

docker-compose exec django bash

Once in, we can make the migrations if it’s the first time we create the app, as we won’t have any.

python manage.py makemigrations

And also run them

python manage.py migrate

If you want to create an admin user to log in into Django’s admin panel,

python manage.py createsuperuser

That’s it!

Now just open your browser and type

http://127.0.0.1:8000/
http://127.0.0.1:8000/admin/

You’re already set up to develop, so happy coding with docker!

Case 2: I already have a Django project

Step1: Clone the docker-django repository which has the docker configuration files.

git clone https://github.com/joeymasip/docker-django.git

Step2: Move all files to your already created Django project.

1. The docker folder containing python + django and a MySQL container config for it.
2. The docker-compose.yml file
3. The .env file

Step3: Start the docker images inside your Django project folder.

cd into your Django project folder and type the following command

docker-compose up -d

This command starts the containers. The parameter -d makes them run in the background. If you omit the -d you’ll see the log.

Docker should start building (if it’s the first time for these images) and running with the containers in the background.

Step4: Update database settings.

So if you’re using the docker-compose.yml out of the box, your database name, user and pw need updating in your project’s settings.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'docker_django_db',
        'USER': 'dbuser',
        'PASSWORD': 'dbpw',
        'HOST': 'mysql',
        'PORT': '3306',
        'TEST': {
            'NAME': 'docker_django_db_test',
        },
    }
}

Step5: Run migrations.

docker-compose exec django bash

Once in, we can run the migrations.

python manage.py migrate

Now, open a new chrome tab and type the following URL. The port is the one we set up in the docker-compose.yml

http://127.0.0.1:8000/
http://127.0.0.1:8000/admin/
http://127.0.0.1:8000/whatever-slug-you-want-from-your-project

You should see it working.

You’re already set up to develop, so happy coding with docker!

Security and Voters in Symfony 3.4

So the other day I was talking to one of our interns on the topic why it’s important to think about security and permissions when writing a backend app, and so I thought, why not write a blog post talking about it.

If you’ve coded a little bit with Symfony, you probably already know that the security component is very useful to secure your app in many ways. There’s different concepts we need to understand before we can talk about voters.

Authentication

The first part to think of when coding a backend app is authentication.

Wether it’s through a login page or a cookie header within the request, the user will have to be authenticated to access the secured part of your application. This way you will prevent anonymous users from accessing it.

How to do this is up to you, but there’s several bundles that help you get started. The most used one is FOSUserBundle. Alternatively, you can code your own custom authentication system and User entities, implementing Symfony’s UserInterface, so you can handle all the roles, etc. More info on implementing a custom authentication system here.

One way or the other, you’ll have to define roles in your application, specifically in role_hierarchy and access_control keys in your security.yml. Let’s look into these.

So for instance, let’s imagine a case where we have an art application, where artists upload their art.

Our security.yml role hierarchy could look something like this:

#app/config/security.yml
security:
	#...
    role_hierarchy:
        ROLE_ARTIST:      ROLE_USER
        ROLE_ADMIN:       [ROLE_ARTIST, ROLE_USER]
        #...

Once you have your authentication firewall up and running, it’s all about the authorization.

Authorization

This is when an already authenticated user is allowed or not into a specific part of an app.

For example, our ACL could look something like this:

#app/config/security.yml
security:
	#...
    access_control:	
        - { path: ^/login$, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/register, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/resetting, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/admin, role: ROLE_USER }

This means that all URIs under /admin will need some type of authentitication.

Let’s imagine we defined a CRUD page with all artists on it, with the following uri in our controller: /admin/artist

So, right now, a user that has ROLE_USER might be able to see a CRUD page with all the users in it! This is wrong! Only a user with ROLE_ADMIN should be able to see that page right? To achieve this we can call the AuthenticateCheker service from Symfony’s security component, like so:

use Symfony\Component\HttpFoundation\Request;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;

/**
 * @Route("/admin/artist")
 */
class ArtistController extends Controller
{
	/**
	 * @Route("/")
	 */
	public function indexAction(Request $request)
	{
		if (false === $this->get('security.authorization_checker')->isGranted('ROLE_ADMIN')) {
	         throw $this->createAccessDeniedException('Unable to access this page!');
	    }
	    // ...
	}
}

Or with Symfony’s base Controller’s wrapper function

use Sensio\Bundle\FrameworkExtraBundle\Configuration\Security;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;

/**
 * @Route("/admin/artist")
 */
class ArtistController extends Controller
{
	/**
	 * @Route("/")
	 */
	public function indexAction(Request $request)
	{
		$this->denyAccessUnlessGranted('ROLE_ADMIN', null, 'Unable to access this page!');
		// ...
	}
}

You can even write the annotation, and the controller will be even thinner!

use Sensio\Bundle\FrameworkExtraBundle\Configuration\Security;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;

/**
 * @Route("/admin/artist")
 */
class ArtistController extends Controller
{
	/**
	 * @Route("/")
	 * @Security("has_role('ROLE_ADMIN')")
	 */
	public function indexAction(Request $request)
	{
		// ...
	}
	// ...
}

Ok, so far so good. We now have a page which is only allowed for admins to access, great!

Let’s talk about an artist being able to edit it’s own profile page.

use Sensio\Bundle\FrameworkExtraBundle\Configuration\Security;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;

/**
 * @Route("/admin/artist")
 */
class ArtistController extends Controller
{
	// ...

	/**
	 * @Route("/edit/{id}")
	 * @Security("has_role('ROLE_ARTIST')")
	 */
	public function editAction(Request $request, Artist $artist)
	{
		// ...
	}
	// ...
}

So we’ve just updated the role and that should do right? Wrong!

What about users who have the same roles, but are not allowed to see each others’ information!!

So for instance, the artist with id 34 should be able to access the URI the /admin/artist/edit/34 but should not be able to access /admin/artist/edit/35, /admin/artist/edit/36,… etc.

Well, that’s where voters come in. This is such a key part of the application and it’s important you take it into account. Do not just secure your actions with roles, secure it with voters. The last thing you want is to have security holes in your application!

So in our previous example, artist with id 34 should only be able to access /admin/artist/edit/34 right? Let’s see how it can be done.

Security Voters

Voters are the services that decide which user can see or access what page. Conceptually, they are called voters because they vote, just like a Parliament or the Government, if a user can access certain information. They vote in favour (ACCESS_GRANTED), against (ACCESS_DENIED) or they abstain themselves (ACCESS_ABSTAIN) when they do not have enough information.

How to build a Voter

A Voter has to implement the VoterInterface. You can implement it yourself, or you can extend the abstract class Voter, which already has some logic into it.

This is what an Artist Voter would look like in our example.


namespace AppBundle\Security;

use AppBundle\Entity\Artist;
use AppBundle\Entity\User;
use Symfony\Component\Security\Core\Authentication\Token\TokenInterface;
use Symfony\Component\Security\Core\Authorization\Voter\Voter;
use Symfony\Component\Security\Core\Authorization\AccessDecisionManagerInterface;

class ArtistVoter extends Voter
{
    // these strings are just invented: you can use anything
    const VIEW = 'view';
    const EDIT = 'edit';

    private $decisionManager;

    public function __construct(AccessDecisionManagerInterface $decisionManager)
    {
        $this->decisionManager = $decisionManager;
    }

    protected function supports($attribute, $subject)
    {
        // if the attribute isn't one we support, return false
        if (!in_array($attribute, array(self::VIEW, self::EDIT))) {
            return false;
        }

        // only vote on Artist objects inside this voter
        if (!$subject instanceof Artist) {
            return false;
        }

        return true;
    }

    protected function voteOnAttribute($attribute, $subject, TokenInterface $token)
    {
        $user = $token->getUser();

        if (!$user instanceof User) {
            // the user must be logged in; if not, deny access
            return false;
        }

        // ROLE_SUPER_ADMIN can do anything! The power!
        if ($this->decisionManager->decide($token, array('ROLE_ADMIN'))) {
            return true;
        }

        // you know $subject is a Artist object, thanks to supports
        /** @var Artist $artist */
        $artist = $subject;

        switch ($attribute) {
            case self::VIEW:
                return $this->canView($artist, $user);
            case self::EDIT:
                return $this->canEdit($artist, $user);
        }

        throw new \LogicException('This code should not be reached!');
    }

    private function canView(Artist $artist, User $user)
    {
        // if they can edit, they can view
        if ($this->canEdit($artist, $user)) {
            return true;
        }

        // the Artist object could have, for example, a method isPrivate()
        // that checks a boolean $private property
        return $user === $artist->getUser();
    }

    private function canEdit(Artist $artist, User $user)
    {
        // this assumes that the data object has a getOwner() method
        // to get the entity of the user who owns this data object
        return $user === $artist->getUser();
    }
}

So the code is pretty self explanatory. There are two functions which we must implement when extending the Voter,

protected function supports($attribute, $subject)

and

protected function voteOnAttribute($attribute, $subject, TokenInterface $token)

.

The other two functions are private functions that are called in the above two. The private functions contain the logic. In our case, it’ll return true if an user is the owner of the artist entity, and false if anything else. Also, if the user has an admin role, it’ll also return true.

The supports function is telling that this Artist Voter will just vote about Artist entities, nothing else. And it’ll also vote on ‘edit’ and ‘view’ attributes, nothing else. If subject and attributes match, it’ll return the result of the vote, which will be a true or false on this case, as it’s what we need in the if section of the

public function vote(TokenInterface $token, $subject, array $attributes)

function of the abstract Voter.

Now let’s use this voter in our artist edit action:

use Sensio\Bundle\FrameworkExtraBundle\Configuration\Security;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;

/**
 * @Route("/admin/artist")
 */
class ArtistController extends Controller
{
	// ...

	/**
	 * @Route("/edit/{id}")
	 * @Security("has_role('ROLE_ARTIST')")
	 */
	public function editAction(Request $request, Artist $artist)
	{
		$this->denyAccessUnlessGranted('edit', $artist);
		// ...
	}
	// ...
}

That would now protect our URI from users who are trying to edit someone else’s page.

Let’s look a bit more into the the wrapper function

$this->denyAccessUnlessGranted('edit', $artist)

from Symfony’s Controller, which uses Symfony’s ControlTrait.

namespace Symfony\Bundle\FrameworkBundle\Controller;

trait ControllerTrait
{
	//...

	/**
     * Throws an exception unless the attributes are granted against the current authentication token and optionally
     * supplied subject.
     *
     * @param mixed  $attributes The attributes
     * @param mixed  $subject    The subject
     * @param string $message    The message passed to the exception
     *
     * @throws AccessDeniedException
     *
     * @final since version 3.4
     */
    protected function denyAccessUnlessGranted($attributes, $subject = null, $message = 'Access Denied.')
    {
        if (!$this->isGranted($attributes, $subject)) {
            $exception = $this->createAccessDeniedException($message);
            $exception->setAttributes($attributes);
            $exception->setSubject($subject);

            throw $exception;
        }
    }

    //...
}

So basically, the second parameter is the subject, AKA, the object we’re trying to edit.

So calling the above function

$this->denyAccessUnlessGranted('edit', $artist);

is the same as calling

$this->get('security.authorization_checker')->isGranted('edit', $artist)

like so:

use Sensio\Bundle\FrameworkExtraBundle\Configuration\Security;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Bundle\FrameworkBundle\Controller\Controller;

/**
 * @Route("/admin/artist")
 */
class ArtistController extends Controller
{
	// ...

	/**
	 * @Route("/edit/{id}")
	 * @Security("has_role('ROLE_ARTIST')")
	 */
	public function editAction(Request $request, Artist $artist)
	{
		if (false === $this->get('security.authorization_checker')->isGranted('edit', $artist)) {
         	throw $this->createAccessDeniedException('Unable to access this page!');
    	}
		// ...
	}
	// ...
}

So as you can see, we’re calling the same function as at the beggining when we wanted to secure the indexAction with the function

$this->get('security.authorization_checker')->isGranted(...)

except this time, we’re seding a second parameter, the subject.

What this piece of code will do is trigger the hook of all voters that implement VoterInterface in your application and try to vote.

And if we look a bit deeper what the vote function does, you can find this code in the abstract class Voter, which we just extended to create our ArtistVoter.

namespace Symfony\Component\Security\Core\Authorization\Voter;

use Symfony\Component\Security\Core\Authentication\Token\TokenInterface;

/**
 * Voter is an abstract default implementation of a voter.
 *
 * @author Roman Marintšenko 
 * @author Grégoire Pineau 
 */
abstract class Voter implements VoterInterface
{

    /**
     * {@inheritdoc}
     */
    public function vote(TokenInterface $token, $subject, array $attributes)
    {
        // abstain vote by default in case none of the attributes are supported
        $vote = self::ACCESS_ABSTAIN;

        foreach ($attributes as $attribute) {
            if (!$this->supports($attribute, $subject)) {
                continue;
            }

            // as soon as at least one attribute is supported, default is to deny access
            $vote = self::ACCESS_DENIED;

            if ($this->voteOnAttribute($attribute, $subject, $token)) {
                // grant access as soon as at least one attribute returns a positive response
                return self::ACCESS_GRANTED;
            }
        }

        return $vote;
    }

    //...
}

In our example, if it supports the attribute ‘edit’ and the subject, in this case, an Artist entity, it’ll try to grant or deny the access. If it doesn’t support this attribute OR this subject, it’ll abstain itself and move on to the next voter.

The great thing about this is you can implement different voters with one same subject but different logic, and in the end, all votes will be counted and the application will decide if that user is granted or not the access, much like a democratic parliament 🙂

Hope this info is useful to understand a bit more what goes under the hood in Symfony’s security component, and how to make sure your app is always secure.

Happy coding!! 🙂

Authentication here.
Authorization here.
Security component here.
Security configuration here.
Voters documentation here.

Symfony 3.4 PHPUnit testing database data

Hey guys, this post is a follow up from my last phpunit in Symfony2 post. This time I’ve had to install a test environment in a Symfony 3.4 project and I’ve had some weird problems, so I thought of writing this blog post once I solved them all, it may be helpful to someone out there 🙂

Step 1 – Installing a fresh Symfony 3.4 project.

In your symfony project, run the composer require

symfony new my_project 3.4

Now run the phpunit command to run tests.

cd my_project
# if you have the phpunit phar installed globally
phpunit tests/
# if you use the phar file directly
php phpunit.phar tests/

Right off the bat, it’ll give you an error…

PHPUnit 5.7.21 by Sebastian Bergmann and contributors.

E                                                                   1 / 1 (100%)

Time: 128 ms, Memory: 14.00MB

There was 1 error:

1) Tests\AppBundle\Controller\DefaultControllerTest::testIndex
Error: Call to undefined method Symfony\Component\Yaml\Parser::parseFile()

This problem is due to an incompatibility with code dependencies from PHPUnit and Symfony 3.2+ code, specifically with the Yaml component.

Don’t panic though, PHPUnit is working to solve this issue.

More info on the PHPUnit issue here.
More info on Symfony’s explanation here.

In the meantime, we can use the PHPUnit bridge to test our application.

cd my_project
composer require --dev symfony/phpunit-bridge

Once your composer json and lock have updated, it’s time to run the test!

./vendor/bin/simple-phpunit

It works!!

Step 2 – Installing doctrine’s fixtures bundle.

First run the composer require

composer require --dev doctrine/doctrine-fixtures-bundle

Once your composer json and lock have updated, it’s time to install it in our Kernel.

// app/AppKernel.php

// ...
if (in_array($this->getEnvironment(), ['dev', 'test'], true)) {
    // ...
    $bundles[] = new Doctrine\Bundle\FixturesBundle\DoctrineFixturesBundle();
}

Now it’s installed, let’s create our first fixture. Let’s suposse we installed FOSUserBundle to manage our users, so for instance we could create a couple users like so:

// AppBundle/DataFixtures/ORM/LoadUserData.php

namespace AppBundle\DataFixtures\ORM;

use Doctrine\Common\DataFixtures\FixtureInterface;
use Doctrine\Common\DataFixtures\OrderedFixtureInterface;
use Doctrine\Common\Persistence\ObjectManager;
use Symfony\Component\DependencyInjection\ContainerAwareInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;
use AppBundle\Entity\User;

class LoadUserData implements FixtureInterface, ContainerAwareInterface
{
    private $data = [
        'ROLE_USER' => [
            'username' => 'user',
            'email' => 'user@slowcode.io',
            'plainPassword' => 'user@slowcode.io'
        ],
        'ROLE_ADMIN' => [
            'username' => 'admin',
            'email' => 'admin@slowcode.io',
            'plainPassword' => 'admin@slowcode.io'
        ],
    ];

    private $container;

    public function setContainer(ContainerInterface $container = null)
    {
        $this->container = $container;
    }

    public function load(ObjectManager $manager)
    {
        $userManager = $this->container->get('fos_user.user_manager');

        foreach ($this->data as $role => $attrs) {
            $user = $userManager->createUser();
            foreach ($attrs as $attr => $val) {
                $function = 'set'. ucwords($attr);
                $user->$function($val);
                $user->setEnabled(true);
            }
            $user->setRoles(['ROLE_USER', $role]);
            $userManager->updateUser($user, true);
        }
    }
}

Now if we run the fixtures like so

php bin/console doctrine:fixtures:load --append

We’ll get another error:

In LoadDataFixturesDoctrineCommand.php line 95:
                                                
  [InvalidArgumentException]                    
  Could not find any fixture services to load. 

To fix this, we’ll need to add this to services.yml

# services.yml
services:
    # ...
    AppBundle\DataFixtures\:
        resource: '../../src/AppBundle/DataFixtures'
        tags: ['doctrine.fixture.orm']

Yes, you guessed it, the magic here relies on the tag. Doctrine will load all services that have that ‘doctrine.fixture.orm’ tag. On the other hand, it’ll also load all the classes that implement the ORMFixtureInterface (for instance, those classes extending the Fixture class). More info on all of this in the DoctrineFixtureBundle doc page.

Step 3 – Setting a separate database for testing.

Add these lines on your config_test.yml file.

# config_test.yml
# ...
# Doctrine Configuration
doctrine:
    dbal:
        driver:   "%test_database_driver%"
        host:     "%test_database_host%"
        port:     "%test_database_port%"
        dbname:   "%test_database_name%"
        user:     "%test_database_user%"
        password: "%test_database_password%"

This will point to the new test database.

Then add the data in parameters.yml (don’t forget parameters.yml.dist for future installs)

# parameters.yml
parameters:
    # ...
    test_database_driver: pdo_mysql
    test_database_host: localhost
    test_database_port: null
    test_database_name: test_db_name
    test_database_user: test_db_user
    test_database_password: test_db_pw

Step 4 – Creating a TestCase that we will extend when ddbb fixtures are needed.

This is useful for reusing code, instead of writing it on every file. It also makes all the fixture tests more consistent.

 //tests/AppBundle/DataFixtures/DataFixtureTestCase.php
namespace Tests\AppBundle\DataFixtures;

use Doctrine\ORM\EntityManager;
use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;
use Symfony\Bundle\FrameworkBundle\Console\Application;
use Symfony\Component\Console\Input\StringInput;
use Symfony\Bundle\FrameworkBundle\Client;
use Symfony\Component\DependencyInjection\ContainerInterface;

class DataFixtureTestCase extends WebTestCase
{
    /** @var  Application $application */
    protected static $application;

    /** @var  Client $client */
    protected $client;
    
    /** @var  ContainerInterface $container */
    protected $container;

    /** @var  EntityManager $entityManager */
    protected $entityManager;

    /**
     * {@inheritDoc}
     */
    public function setUp()
    {
        self::runCommand('doctrine:database:drop --force');
        self::runCommand('doctrine:database:create');
        self::runCommand('doctrine:schema:create');
        self::runCommand('doctrine:fixtures:load --append --no-interaction');

        $this->client = static::createClient();
        $this->container = $this->client->getContainer();
        $this->entityManager = $this->container->get('doctrine.orm.entity_manager');

        parent::setUp();
    }

    protected static function runCommand($command)
    {
        $command = sprintf('%s --quiet', $command);

        return self::getApplication()->run(new StringInput($command));
    }

    protected static function getApplication()
    {
        if (null === self::$application) {
            $client = static::createClient();

            self::$application = new Application($client->getKernel());
            self::$application->setAutoExit(false);
        }

        return self::$application;
    }
    
    /**
     * {@inheritDoc}
     */
    protected function tearDown()
    {
        self::runCommand('doctrine:database:drop --force');

        parent::tearDown();

        $this->entityManager->close();
        $this->entityManager = null; // avoid memory leaks
    }
}

This will create the database and install the fixtures on every test that extends this TestCase. This is important as you want a consistent database with the same data every time!

Step 5 – Creating the unit, functional and integration tests

After creating your test fixtures, now you can test your services logic easily by extending the TestCase we just created, like so.

// tests/AppBundle/User/FooTest.php

class FooTest extends DataFixtureTestCase
{
    protected $fooService;

    /**
     * {@inheritDoc}
     */
    public function setUp()
    {
        parent::setUp();
        $this->fooService = $this->container->get('app.service.foo');
    }

    public function test_get_two_users()
    {
        $users = $this->entityManager->getRepository(User::class)->findAll();
        $this->assertEquals(2, count($users));
    }

  
}

Now you just need to run the phpunit command like so

./vendor/bin/simple-phpunit

I’ve created a bash wrapper script called ‘test’ that contains just this line of code so now I just need to run

bash test

Happy coding and testing! 🙂

Translatable models for translatable website in Django

This post is about using model translations (without translation files) and showing some ways to send that translations to the front end where they can be rendered.

This post is not about installing modeltranslations in Django, there’s a comprehensive guide on how to do that here.

Study case: Translatable texts that need to be editable in the django admin backend, without translation files.

Once we’ve got the modeltranslations module installed, we need to create the model we want to translate.

I’ve created the model String, but you can call it whatever you want.

#your_app/models/string.py
from django.db import models

class String(models.Model):
    key = models.CharField(max_length=80)
    value = models.CharField(max_length=255, blank=True, null=True)

    def __str__(self):
        return self.key

Now we should register our model in the translation.py, and so we can specify what field(s) will be the translatable one.

#your_app/translation.py
class StringTranslationOptions(TranslationOptions):
    fields = ('value',)


translator.register(String, StringTranslationOptions)

In a database level, this model table called your_app_string will contain the translations for every key in a column for each language: value_en, value_es, …

If you have i18n patterns in your url settings, Django will automatically know what locale to serve you through it’s own middleware system. For more info on this, there’s a comprehensive guide here.

Otherwise, you can set the locale manually through the request, up to you. More info on this here.

Once Django already knows what’s the current locale, we just need to get the strings from the database. The modeltranslations module will automatically give us the value translated to the current locale.

Translating the front end

We are able to transfer the strings to the frontend templates through many ways. For instance, we could load the strings on every view before rendering the template. However, this is bit of a code smell, as code is too coupled and it’s not very mantainable. We can do better like so:

1. Using a Context Processor

#your_app/context_processors/trans.py
from your_app.models import String


def trans(request):
    trans = {}
    for string in String.objects.all():
        trans[string.key] = string.value

    return {'trans': trans}

Your’ll need to register the context processor in settings.py

#settings.py
TEMPLATES = [
    {	
    	'...',
        'OPTIONS': {
            'context_processors': [
            	'...',
                'your_app.context_processors.trans',
            ],
        },
    },
]

What this will do is, after every render function in our views, it’ll include the array of all translations. That’s why you don’t need to load any extra file in the template.

Now in the front end, you can use it like so:

{{ trans.hello_world }}

{{ trans.hello_world_wrong_key }}

This is similar to loading the strings just before every render in the views, however we’ve removed the coupling by centralizing the code in one place.

The main problem though is that if there’s a key that’s not included, Django will just ignore that key, not giving us any feedback if a translation is missing. This is annoying if we’re developing, as we won’t catch missing translations for certain keys.

2. Using a filter (templatetag)

#your_app/templatetags/trans.py
@register.filter(name='trans')
def trans(field):

    translations = {}
    for string in String.objects.all():
        translations[string.key] = string.value

    try:
        trans = translations[field.__str__()]
    except KeyError:
        trans = '[*' + field + '*]'

    return trans

Now in the front end, you can use it like so. This time, different from context_processor, we won’t need to register it in the settings.py but we do need to load it in the template.

{% load trans %}

{{ 'hello_world' | trans }}

{{ 'hello_world_wrong_key' | trans }}

This one gives feedback if the key doesn’t exist, but as you can see, it loads all the strings every time it finds a key to be rendered. This is very inefficient!

3. Using a simple tag (templatetag)

Very similar to the filter, but there’s a suttle difference, this one can load the context.

#your_app/templatetags/trans.py
@register.simple_tag(name='trans', takes_context=True)
def trans(context, field):
    translations = context.request.session.get('translations')

    if not translations:
        translations = {}
        for string in String.objects.all():
            translations[string.key] = string.value
        context.request.session['translations'] = translations

    try:
        trans = translations[field.__str__()]
    except KeyError:
        trans = '[*' + field + '*]'

    return trans

Loading the context allows us to save data in memory, so it doesn’t have to load the translations from the database every single time there’s a string to be translated.

In the front end we still need to load the tag.

{% load trans %}

{% trans 'hello_world' %}

{% trans 'hello_world_wrong_key' %}

This one is not only clean and efficient, it also gives feedback if a translation key doesn’t exist in the database. Furthermore, since we now have the context (it can’t be loaded with a filter), the translations will be loaded from the ddbb just once (when it tries to translate the first key). When there’s a second key to be translated, it’ll get the array of translations from the context and just give the value translated, or the key if it can’t find it.

So there you have it, three different ways to present the translations in the front end with a custom model translation.

Other useful resources:

Django translations
Django model translations

How to add attributes to form widgets in Django templates

This blog post is about templates in Django, specifically filters or pipes, whatever you call them – in Symfony we used to call them Twig Filters – and how to create them.

Let’s put I have to add a class “form-control” from bootstrap, and another class “select2” from the select2 library into a form widget. Since the form widget is rendered in the backend, it’s not clear how to add attributes before it’s rendered. In this example I had to create a pipe to add attributes to the form widget.

In Symfony’s Twig, it’s pretty simple to add attributes, like so:

{{ form_widget(form.start, {'attr': {'class': 'form-control select2'}} ) }}

In Django’s template we’ll need a bit of a workaround and create the pipe.

Creating the filter file, add_attr.py

from django import template
register = template.Library()


@register.filter(name='add_attr')
def add_attr(field, css):
    attrs = {}
    definition = css.split(',')

    for d in definition:
        if ':' not in d:
            attrs['class'] = d
        else:
            key, val = d.split(':')
            attrs[key] = val

    return field.as_widget(attrs=attrs)

This file should live in a folder called templatetags in your Django app.

Using the pipe in the template

To use it in a Django template, firstly add this widget in the top of the template, similar to the load static widget.

{% load add_attr %}

And now you can use it on any form widget as a function, like so

{{ form.start|add_attr:"class:form-control select2" }}

And that’s it! Your widget will have the classes form-control and select2.

Hope this helps. Alternatively, you can use a django library called django-widget-tweaks that helps you to do this, find the info here.

Happy coding! 🙂