Python Console App with Docker

Today I want to write about dockerizing a python app, for instance, a python script.

Let’s imagine we want to run this script in different machines and we can’t control what python version they have, or even worse, we don’t know if python is installed or not, and what dependencies they have installed.

Docker helps us separate the code from the infrastructure so this blog post will be about this.

Create the Python script

Let’s create a simple python script. We’ll call the file script.py inside a src folder of the project.

#src/script.py

print('hello world!')

Create the Dockerfile

#Dockerfile

FROM python:3.6

RUN mkdir /application
WORKDIR "/application"

# Upgrade pip
RUN pip install --upgrade pip

# Update
RUN apt-get update \
    && apt-get clean; rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* /usr/share/doc/*

ADD requirements.txt /application/
ADD src/script.py /application/

RUN pip install -r /application/requirements.txt

CMD [ "python", "script.py" ]

FROM tells Docker which image you base your image on. In the example, Python 3.

RUN tells Docker which additional commands to execute in bash console. In the example, we create a folder called application and we later install the requirements with the pip library we installed earlier.

CMD tells Docker to execute the command when the image loads. In the example, our python script.

How to run

In your console, build the docker image

docker build -t python-script .

And then run it

docker run python-blog-scraper

You should now get a ‘hello world’ printed 🙂

Requirements

You might ask yourself, why not just run the script, i.e python script.py from the console? My answer to that is, because of the requirements. As I mentioned earlier, systems change and it’s not realistic to think that you’ll always have the python version you need with the pip requirements installed.

So let’s test this out with some requirements.

#requirements.txt

requests
beautifulsoup4

Now let’s use these requirements in our script. For instance, let’s make a GET request to my blog and let’s scrape the contents of it, maybe the title of the blog posts and the first sentence. Print these in our console.

Note: Scraping is better done with Selenium in my opinion, but just as an example, let’s just use beatifulsoup’s library.

#blog-scraper.py
import requests
from bs4 import BeautifulSoup


class BlogPost:
    title = None
    first_sentence = None

    def __init__(self, title, first_sentence):
        self.title = title
        self.first_sentence = first_sentence

    def __str__(self):
        return str(self.title) + '\n' + str(self.first_sentence)


def scrape(url):
    r = requests.get(url)

    if r.status_code == 200:
        soup = BeautifulSoup(r.content, features="html.parser")
        # print(soup.prettify())

        for article in soup.find_all('article', attrs={'class': 'post'}):
            title = article.find('h2', attrs={'class': 'entry-title'}).string
            first_sentence = article.find('div', attrs={'class': 'entry-content'}).find('p').string

            blog_post = BlogPost(
                title=title,
                first_sentence=first_sentence
            )
            print(blog_post)
            print('------------------------------')
    else:
        print('Whoops! Something went wrong!')


scrape("https://blog.joeymasip.com/")

For readability, I created a model that has the information we want to keep to print later. We’re printing as we run the loop here, not saving anything to an array or database (this is just as an example so we can use libraries we just installed). Also, I renamed this blog-scraper.py, so don’t forget to update your Dockerfile!

#Dockerfile
#...
ADD requirements.txt /application/
ADD src/blog-scraper.py /application/

RUN pip install -r /application/requirements.txt

CMD [ "python", "blog-scraper.py" ]
#...

Now let’s rebuild the image and run it.

docker build -t python-script .
docker run python-blog-scraper

And there we have it, you should have the blog posts being printed in your console!

Conclusion and Resources

As you can see, we’ve added requirements and packaged everything in a Docker image, so we can easily deploy this simple app to any machine that has docker installed and run it with no requirement missing or version issues.

I’ve uploaded the source code in GitHub here.
Link to Python library Beautifulsoup docs.
Link to Python library Requests docs.

Happy coding! 🙂

How to create an API with Symfony 4 and JWT

Today we’re going to create a Symfony 4 API web app from scratch – I’ll walk you through all the steps, so by the end of this tutorial, you should be able to create, configure and run a web app with API endpoints and protected with JWT authentication.

Also, I’ve uploaded all the source code here so you can follow through the tutorial or you can download the code and play with it while you read.

1. Docker

To set our development environment, we’ll use Docker – you probably already know by now how much I love Docker 🙂

Let’s start by creating a docker-compose.yaml file with php7, mysql for database and nginx for the webserver.

#docker-compose.yaml
version: "3.1"

volumes:
    db-data:

services:
    mysql:
      image: mysql:5.6
      container_name: ${PROJECT_NAME}-mysql
      working_dir: /application
      volumes:
        - db-data:/application
      environment:
        - MYSQL_ROOT_PASSWORD=docker_root
        - MYSQL_DATABASE=sf4_db
        - MYSQL_USER=sf4_user
        - MYSQL_PASSWORD=sf4_pw
      ports:
        - "8306:3306"

    webserver:
      image: nginx:alpine
      container_name: ${PROJECT_NAME}-webserver
      working_dir: /application
      volumes:
        - .:/application
        - ./docker/nginx/nginx.conf:/etc/nginx/conf.d/default.conf
      ports:
        - "8000:80"

    php-fpm:
      build: docker/php-fpm
      container_name: ${PROJECT_NAME}-php-fpm
      working_dir: /application
      volumes:
        - .:/application
        - ./docker/php-fpm/php-ini-overrides.ini:/etc/php/7.2/fpm/conf.d/99-overrides.ini
      environment:
        XDEBUG_CONFIG: "remote_host=${localIp}"

Also, don’t forget the .env file with your PROJECT_NAME variable. On this ocasion, for the php-fpm and the nginx, I’m pointing to docker folder, so I can override the nginx.conf file and the php-fpm Dockerfile with special configuration, such as xdebug.

Link to docker/nginx/nginx.conf file
Link to docker/php-fpm/Dockerfile
Link to docker/php-fpm/php-ini-overrides.ini

Once we’re ready, we can build and run.

docker-compose build
docker-compose up -d

2. Creating a Symfony project

First, let’s go into the bash

docker-compose exec php-fpm bash

Let’s create a symfony 4 project

#inside php-fpm bash
composer create-project symfony/website-skeleton symfony

Clean up

#inside php-fpm bash
mv /application/symfony/* /application
mv /application/symfony/.* /application
rm -Rf /application/symfony

More on how to create a symfony 4 application with docker here

If we open the browser http://localhost:8000/ we should see the Symfony welcome page. So now that we have a symfony 4 app up and running, let’s start putting stuff into it!

3. Mapping our User in the database

Inside the php-fpm bash, let’s start with installing the FOSUserBundle to have a User base entity we can relate to.

#inside php-fpm bash
composer require friendsofsymfony/user-bundle "~2.0"
Note: you'll probably get the error "The child node "db_driver" at path "fos_user" must be configured." Don't panic, this is unfortunately normal. Reason being is it's trying to clear the cache before configuration is correct.

3.1 Configuriation

#config/services.yaml

# FOS user config
fos_user:
    db_driver:      orm # other valid values are 'mongodb', 'couchdb' and 'propel'
    firewall_name:  main
    user_class:     App\Entity\User
    from_email:
        address: "no-reply@joeymasip.com"
        sender_name: "Joey"
    registration:
#        form:
#            type: AppBundle\Form\UserRegisterType
        confirmation:
            enabled: true
            template:   FOSUserBundle:Registration:email.txt.twig
            from_email:
                address:        "no-reply@joeymasip.com"
                sender_name:    "No Reply Registration"
    service:
        mailer: fos_user.mailer.twig_swift
    resetting:
        email:
            template:   FOSUserBundle:Resetting:email.txt.twig

3.2 Creating the User class

namespace App\Entity;

use FOS\UserBundle\Model\User as BaseUser;
use Doctrine\ORM\Mapping as ORM;

/**
 * @ORM\Entity
 * @ORM\Table(name="fos_user")
 */
class User extends BaseUser
{
    /**
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    protected $id;

    public function __construct()
    {
        parent::__construct();
        // your own logic
    }
}

3.3 Configuring main firewall

#config/packages/security.yaml
security:
    encoders:
        FOS\UserBundle\Model\UserInterface: bcrypt
        Symfony\Component\Security\Core\User\User: plaintext

    role_hierarchy:
        ROLE_ADMIN:         ROLE_USER
        ROLE_SUPER_ADMIN:   ROLE_ADMIN

    providers:
        chain_provider:
            chain:
                providers: [in_memory, fos_userbundle]
        in_memory:
            memory:
                users:
                    superadmin:
                        password: 'superadminpw'
                        roles: ['ROLE_SUPER_ADMIN']
        fos_userbundle:
            id: fos_user.user_provider.username

    access_control:
        - { path: ^/login$, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/register, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/resetting, role: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/admin/, role: ROLE_ADMIN }

    firewalls:
        dev:
            pattern: ^/(_(profiler|wdt)|css|images|js)/
            security: false
        main:
            pattern: ^/
            form_login:
                provider: chain_provider
                csrf_token_generator: security.csrf.token_manager
                login_path: fos_user_security_login
                check_path: fos_user_security_check
                always_use_default_target_path: false
                default_target_path: admin_admin_index

            logout:
                path:   fos_user_security_logout
                target: fos_user_security_login
            anonymous:    true

3.4 Creating the register API endpoint

Now that we have a User entity mapped in our database, let’s create a register API endpoint so we can add new users.

I’ve created an Api folder, and added in routes.yaml

#config/routes.yaml
api:
    prefix: /api
    resource: '../src/Controller/Api'

So all our API endpoints will have the prefix api

So now let’s create a Controller for registering our users.

namespace App\Controller\Api;

use FOS\UserBundle\Model\UserManagerInterface;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;
use App\Entity\User;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Component\Validator\Validation;
use Symfony\Component\Validator\Constraints as Assert;

/**
 * @Route("/auth")
 */
class ApiAuthController extends AbstractController
{
    /**
     * @Route("/register", name="api_auth_register",  methods={"POST"})
     * @param Request $request
     * @param UserManagerInterface $userManager
     * @return JsonResponse|\Symfony\Component\HttpFoundation\RedirectResponse
     */
    public function register(Request $request, UserManagerInterface $userManager)
    {
        $data = json_decode(
            $request->getContent(),
            true
        );

        $validator = Validation::createValidator();

        $constraint = new Assert\Collection(array(
            // the keys correspond to the keys in the input array
            'username' => new Assert\Length(array('min' => 1)),
            'password' => new Assert\Length(array('min' => 1)),
            'email' => new Assert\Email(),
        ));

        $violations = $validator->validate($data, $constraint);

        if ($violations->count() > 0) {
            return new JsonResponse(["error" => (string)$violations], 500);
        }

        $username = $data['username'];
        $password = $data['password'];
        $email = $data['email'];

        $user = new User();

        $user
            ->setUsername($username)
            ->setPlainPassword($password)
            ->setEmail($email)
            ->setEnabled(true)
            ->setRoles(['ROLE_USER'])
            ->setSuperAdmin(false)
        ;

        try {
            $userManager->updateUser($user, true);
        } catch (\Exception $e) {
            return new JsonResponse(["error" => $e->getMessage()], 500);
        }

        return new JsonResponse(["success" => $user->getUsername(). " has been registered!"], 200);
    }
}
Note: validation and data handling for user creation should be decoupled from the controller, it has been put together just for the example.

If you now send a POST request with the data to http://localhost:8000/api/auth/register, you should get a registered user, and validation error if the data in the json is incorrect or the keys some keys are missing. Also, you’ll get a doctrine error if the username or emails you try to add in the database already exist, as they are unique keys in the FOSUserBundle base User we’re using.

curl -X POST -H "Content-Type: application/json" http://localhost:8000/api/auth/register -d '{"username":"patata","password":"fregida", "email":"patatafregida@joeymasip.com"}'

4. LexikJWTAuthenticationBundle

Now it’s time for the login and recieving a token. For this, we’ll use JWT.

#inside php-fpm bash
composer require lexik/jwt-authentication-bundle

4.1 Private and Public keys

First, let’s create the private and public keys for our project, with a passphrase.

#inside php-fpm bash
mkdir config/jwt
openssl genrsa -out config/jwt/private.pem -aes256 4096
openssl rsa -pubout -in config/jwt/private.pem -out config/jwt/public.pem

4.2 Configuration

Once you’ve created the keys, you can add the config in the yaml and .env files. The passphrase you created the keys with must relate to the config.

#config/packages/lexik_jwt_authentication.yaml
lexik_jwt_authentication:
    secret_key: '%env(resolve:JWT_SECRET_KEY)%'
    public_key: '%env(resolve:JWT_PUBLIC_KEY)%'
    pass_phrase: '%env(JWT_PASSPHRASE)%'
    token_ttl: 3600
#.env
JWT_SECRET_KEY=%kernel.project_dir%/config/jwt/private.pem
JWT_PUBLIC_KEY=%kernel.project_dir%/config/jwt/public.pem
JWT_PASSPHRASE=sf4jwt

4.3 Routes

Once we’ve created the keys and configured the bundle, it’s time to add the login route in our routes.yaml

#config/routes.yaml
api_auth_login:
    path: /api/auth/login
    methods:  [POST]

api:
    prefix: /api
    resource: '../src/Controller/Api'
Note: it's important to put the specific routes before the main ones. See that /api/auth/login is more specific than /api

4.4 Firewalls

And now we have to tell our app to handle this route through configuration, since we won’t be implementing it in our controller.

#config/packages/security.yaml
security:
	#...
    firewalls:
        dev:
            #...

        api_login:
            pattern:  ^/api/auth/login
            stateless: true
            anonymous: true
            json_login:
                provider: chain_provider
                check_path:               /api/auth/login
                success_handler:          lexik_jwt_authentication.handler.authentication_success
                failure_handler:          lexik_jwt_authentication.handler.authentication_failure
            provider: chain_provider

        main:
        	#...

Again, make sure to put api firewalls before the main. The pattern used for the “main” firewall catches everything, the pattern for “api” catches “/api”, so you should put the wildcard AKA main at the end, after the specific cases.

If we now send a POST request to http://localhost:8000/api/auth/login with the username and password from the user we created earlier, you should get a response with the token!

curl -X POST -H "Content-Type: application/json" http://localhost:8000/api/auth/login -d '{"username":"patata","password":"fregida"}'

We’ll get the 200 response

{"token":"eyJ0eXAiOiJKV1QiLCJhbGciOi..."}

All of this is great!

Now it’s time to protect our API calls now that we have tokens right? This is done by creating a new firewall in our security.yaml.

#config/packages/security.yaml
security:
	#...
    firewalls:
        dev:
            #...

        api_login:
            #...

		api:
            pattern: ^/api
            stateless: true
            anonymous: false
            provider: chain_provider
            guard:
                authenticators:
                    - lexik_jwt_authentication.jwt_token_authenticator
        main:
        	#...

Now if we try to call to our previous route http://localhost:8000/api/auth/register without any Authorization header, we’ll see that we get a 401 error.

{
    "code": 401,
    "message": "JWT Token not found"
}

This doesn’t make sense because someone who has to register doesn’t have a token yet! Let’s add one last firewall so anonymous users can register.

#config/packages/security.yaml
security:
	#...
    firewalls:
        dev:
            #...

        api_login:
            #...

        api_register:
            pattern:  ^/api/auth/register
            stateless: true
            anonymous: true

		api:
            #...

        main:
        	#...

To authenticate in our api calls, we just need to add an Authorization header with the Bearer prefix followed by the JWT token, so the value of the header could be for example:

Bearer eyJ0eXAiOiJKV1QiLCJhbGc...

Since we added the anonymous in the /api/auth/register pattern in our firewall, we should now be able to register without sending any token in the header.

Moreover, if you want to open API calls even without a JWT token in the Authorization header, you can just set anonymour to true in the api firewall, like so:

#config/packages/security.yaml
security:
	#...
    firewalls:
        dev:
            #...

        api_login:
            #...

        api_register:
        	#...

		api:
            #...
            anonymous: true
            #...

        main:
        	#...

Now we can add the ACL to control access to fully secure all the prefix routes in our security.yaml

#config/packages/security.yaml
security:
	#...
    access_control:
        #...
        - { path: ^/api/auth/login, roles: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/api/auth/register, roles: IS_AUTHENTICATED_ANONYMOUSLY }
        - { path: ^/api, roles: IS_AUTHENTICATED_FULLY }

One last thing, once we register, it would be nice to receive the token inmediately. It would be wierd for an app to make you register and then make you login afterwards. We can redirect to the /auth/login route once a User has been created and return the response.

So in our previous ApiAuthController,

namespace App\Controller\Api;

/**
 * @Route("/auth")
 */
class ApiAuthController extends AbstractController
{
    /**
     * @Route("/register", name="api_auth_register",  methods={"POST"})
     * @param Request $request
     * @param UserManagerInterface $userManager
     * @return JsonResponse|\Symfony\Component\HttpFoundation\RedirectResponse
     */
    public function register(Request $request, UserManagerInterface $userManager)
    {
        #...

        # Code 307 preserves the request method, while redirectToRoute() is a shortcut method.
        return $this->redirectToRoute('api_auth_login', [
            'username' => $data['username'],
            'password' => $data['password']
        ], 307);
    }
}

5. NelmioApiDocBundle

Let’s add some docs in our API project.

#inside php-fpm bash
composer require nelmio/api-doc-bundle

5.1 Configuration

#config/packages/nelmio_api_doc.yaml
nelmio_api_doc:
    documentation:
#        schemes: [http, https]
        info:
            title: Symfony JWT API
            description: Symfony JWT API docs
            version: 1.0.0
        securityDefinitions:
            Bearer:
                type: apiKey
                description: 'Authorization: Bearer {jwt}'
                name: Authorization
                in: header
        security:
            - Bearer: []
    areas: # to filter documented areas
        path_patterns:
            - ^/api(?!/doc$) # Accepts routes under /api except /api/doc

5.2 Routing

Also uncomment the app.swagger_ui route in

#config/routes/nelmio_api_doc.yaml
app.swagger_ui:
    path: /api/doc
    methods: GET
    defaults: { _controller: nelmio_api_doc.controller.swagger_ui }

5.3 ACL

Finally, we want to add this to our ACL

#config/packages/security.yaml
security:
	#...
    access_control:
        #...
        - { path: ^/api/doc, roles: IS_AUTHENTICATED_ANONYMOUSLY }
        #...
        - { path: ^/api, roles: IS_AUTHENTICATED_FULLY }

Again, remember to put this specific route before the generic /api prefix

If we open the browser http://localhost:8000/api/doc we should see the swagger with the jwt api key auth for secured routes.

6. NelmioCorsBundle

Cross-origin resource sharing is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain from which the first resource was served. So if we want to access our API from a different domain we’ll probably run into CORS problems, so let’s quickly set up NelmioCorsBundle!

#inside php-fpm bash
composer req cors

Change the default configuration from this

#config/packages/nelmio_cors.yaml
nelmio_cors:
    defaults:
        origin_regex: true
        allow_origin: ['%env(CORS_ALLOW_ORIGIN)%']
        allow_methods: ['GET', 'OPTIONS', 'POST', 'PUT', 'PATCH', 'DELETE']
        allow_headers: ['Content-Type', 'Authorization']
        expose_headers: ['Link']
        max_age: 3600
    paths:
        '^/': ~

To this

#config/packages/nelmio_cors.yaml
nelmio_cors:
    defaults:
        allow_credentials: false
        allow_origin: []
        allow_headers: []
        allow_methods: []
        expose_headers: []
        max_age: 0
        hosts: []
        origin_regex: false
        forced_allow_origin_value: ~
    paths:
        '^/api/':
            allow_origin: ['*']
            allow_headers: ['Content-Type', 'Authorization']
            allow_methods: ['POST', 'PUT', 'GET', 'DELETE']
            max_age: 3600
        '^/':
            origin_regex: true
            allow_origin: ['^http://localhost:[0-9]+']
            allow_headers: ['Content-Type', 'Authorization']
            allow_methods: ['POST', 'PUT', 'GET', 'DELETE']
            max_age: 3600
            hosts: ['^api\.']

This will allow all origins for the /api prefix so any mobile app can now use our API. Feel free to play with the regex until you’re comfortable with the result.

7. Creating an example API enpoint

As an example, let’s create an API endpoint for retrieving a user.

I’ll create a new controller

namespace App\Controller\Api;

use App\Entity\User;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
use Symfony\Component\HttpFoundation\JsonResponse;
use Symfony\Component\Routing\Annotation\Route;
use Symfony\Component\Serializer\Serializer;
use Symfony\Component\Serializer\Encoder\XmlEncoder;
use Symfony\Component\Serializer\Encoder\JsonEncoder;
use Symfony\Component\Serializer\Normalizer\ObjectNormalizer;

/**
 * @Route("/user")
 */
class ApiUserController extends AbstractController
{
    /**
     * @Route("/{id}", name="api_user_detail", methods={"GET"})
     * @param User $user
     * @return JsonResponse
     */
    public function detail(User $user)
    {
        $this->denyAccessUnlessGranted('view', $user);
        return new JsonResponse($this->serialize($user), 200);
    }

    protected function serialize(User $user)
    {
        $encoders = [new XmlEncoder(), new JsonEncoder()];
        $normalizers = [new ObjectNormalizer()];

        $serializer = new Serializer($normalizers, $encoders);

        $json = $serializer->serialize($user, 'json');

        return $json;
    }
}
Note: Please consider serializing in a separate service, this is a just some example code.

We’ll need a voter to let us know if a user is allowed to be retrieved or not. It should only be retrieved if it’s himself, or if it’s an admin right?

namespace App\Security;

use App\Entity\User;
use Symfony\Component\Security\Core\Authentication\Token\TokenInterface;
use Symfony\Component\Security\Core\Authorization\Voter\Voter;
use Symfony\Component\Security\Core\Authorization\AccessDecisionManagerInterface;

class UserVoter extends Voter
{
    // these strings are just invented: you can use anything
    const VIEW = 'view';
    const EDIT = 'edit';

    private $decisionManager;

    public function __construct(AccessDecisionManagerInterface $decisionManager)
    {
        $this->decisionManager = $decisionManager;
    }

    protected function supports($attribute, $subject)
    {
        // if the attribute isn't one we support, return false
        if (!in_array($attribute, array(self::VIEW, self::EDIT))) {
            return false;
        }

        // only vote on User objects inside this voter
        if (!$subject instanceof User) {
            return false;
        }

        return true;
    }

    protected function voteOnAttribute($attribute, $subject, TokenInterface $token)
    {
        $user = $token->getUser();

        if (!$user instanceof User) {
            // the user must be logged in; if not, deny access
            return false;
        }

        // ROLE_SUPER_ADMIN can do anything! The power!
        if ($this->decisionManager->decide($token, array('ROLE_ADMIN'))) {
            return true;
        }

        // you know $subject is a User object, thanks to supports
        /** @var User $userSubject */
        $userSubject = $subject;

        switch ($attribute) {
            case self::VIEW:
                return $this->canView($userSubject, $user);
            case self::EDIT:
                return $this->canEdit($userSubject, $user);
        }

        throw new \LogicException('This code should not be reached!');
    }

    private function canView(User $userSubject, User $user)
    {
        // if they can edit, they can view
        if ($this->canEdit($userSubject, $user)) {
            return true;
        }

        // the User object could have, for example, a method isPrivate()
        // that checks a boolean $private property
        return $user === $userSubject;
    }

    private function canEdit(User $userSubject, User $user)
    {
        // this assumes that the data object has a getOwner() method
        // to get the entity of the user who owns this data object
        return $user === $userSubject;
    }
}

As you can see, we can get the user from the TokenInterface. The function

$tokenInterface->getUser()

will return a UserInterface aka our User. The TokenInterface can be injected in an EventListener, Controller, etc. so you can always know what user is making the request.

More on how to use voters for securing your app here.

Let’s try getting that user with the GET request.

curl -X GET -H "Content-Type: application/json" http://localhost:8000/api/user/1

The response is a 401

{"code":401,"message":"JWT Token not found"}

This time let’s add the token in the header, like so

curl -X GET -H "Content-Type: application/json" -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE1NDgyODI4NjUsImV4cCI6MTU0ODI4NjQ2NSwicm9sZXMiOlsiUk9MRV9VU0VSIl0sInVzZXJuYW1lIjoicGF0YXRhIn0.SLB40jBqtbY7Ql5DI4L4rZBEcH5hXNqn9u0eVYOCtlE_vqa_1RYQzHHVy7iMbmP4CMjSKUiBlGzuIBGApRmD36CFKgdMdfXuqrHeEFB-BUsd-HezdLg6U762GnLbe4g4vHzg9XKZVCzRtpboGUKgVycIaMdfiZ1FvUkJeZvdS_5HgHW43LrSqntPlbEaNaEYv7mrkzTQNi1WiKwJaDCW5M0JgRkfbHoUYXMadFUxR4KSnaXRQAwxZnqqPBW4dRs97ho5A15XKpuxmEWemvhMs0XL9E8KyPNG7ZipLis4JGs3X9Mn-ov4RDCqYbShSNbtj_F2gcakXL97FF3myLn1U2XfIzuwxq9ZIJnGLtemKgPlxSB1uxX5ep9aYxppuXpwxY0vGr9MsOgyL3kkuMqeXvFDN46bY-3P8TLOqEuPrrlKYuRfMQv6Wrhdq0orl3eo7t83YCb_Z-Mf7yeDDGeJsftaj4pALJUw4Ovo6Kv_4gNcG3VQpkJr4XtnULAcO9O_OJLgVOBXoOc7lUQmokdvAGeltEBmYIZD_2KtGrTwS4rL55LMn3MawL4dKVIAg8aaYbPDCxkk1t3LdZyI5zSUiJvLaCrMM8ZhJ7eJ0rKod2-d_dZcYPzQ5RF_wD8spuw1pkT6r4hMyTJvGQUUDjN-3E-MkNBHT8Ku8Z8I7a65x5M" http://localhost:8000/api/user/1

This will return the user serialized, so success!

Conclusion

We’ve created a project from scratch and we’ve installed a User library, a JWT library, an API doc library and a CORS library. We’ve created a register endpoint, we’ve protected our api with firewalls and we’ve created a user endpoint with a voter to test authentication and authorization.

This is a great start to develop your API project and scale it up!

Happy coding! 🙂

Resources

Link to symfony4-api-jwt GitHub project.
Link to all commits.

Docker to set up the development environment.
LexikJWTAuthenticationBundle to authenticate users on our API side.
FOSUserBundle for User entities.
NelmioApiDocBundle for our API docs.
NelmioCorsBundle for Cross Origin.

Gedmo Translations in Symfony 4

Today I’ve had to install Gedmo Translations into a Symfony 4 app and I had some trouble, so after fixing them problems I thought of writing them down in case someone else can benefit from it 🙂

Let’s start fromt he begginig.

Create a Symfony 4 project

Please use docker! 🙂 More info here in the Docker for Symfony 4 post.

Here’s what my docker.compose.yml looks like

#docker.compose.yml
version: "3.1"

volumes:
    db-data:

services:
    mysql:
      image: mysql:5.6
      container_name: ${PROJECT_NAME}-mysql
      working_dir: /application
      volumes:
        - db-data:/application
      environment:
        - MYSQL_ROOT_PASSWORD=docker_root
        - MYSQL_DATABASE=gedmoapp_db
        - MYSQL_USER=gedmoapp_user
        - MYSQL_PASSWORD=gedmoapp_pw
      ports:
        - "8306:3306"

    webserver:
      image: nginx:alpine
      container_name: ${PROJECT_NAME}-webserver
      working_dir: /application
      volumes:
        - .:/application
        - ./docker/nginx/nginx.conf:/etc/nginx/conf.d/default.conf
      ports:
        - "8000:80"

    php-fpm:
      build: docker/php-fpm
      container_name: ${PROJECT_NAME}-php-fpm
      working_dir: /application
      volumes:
        - .:/application
        - ./docker/php-fpm/php-ini-overrides.ini:/etc/php/7.2/fpm/conf.d/99-overrides.ini

Installation of Gedmo’s bundle

composer require stof/doctrine-extensions-bundle

Configuration

Now we need to update our configuration file, so in the doctrine.yaml, under the doctrine key, we should go from this

#doctrine.yaml
doctrine:
#...
    orm:
        auto_generate_proxy_classes: '%kernel.debug%'
        naming_strategy: doctrine.orm.naming_strategy.underscore
        auto_mapping: true
        naming_strategy: doctrine.orm.naming_strategy.underscore
        auto_mapping: true
        mappings:
            App:
                is_bundle: false
                type: annotation
                dir: '%kernel.project_dir%/src/Entity'
                prefix: 'App\Entity'
                alias: App

To this

#doctrine.yaml
doctrine:
#...
	orm:
        auto_generate_proxy_classes: '%kernel.debug%'
#        naming_strategy: doctrine.orm.naming_strategy.underscore
#        auto_mapping: true
        entity_managers:
            default:
#                connection: default
                naming_strategy: doctrine.orm.naming_strategy.underscore
                auto_mapping: true
                mappings:
                    App:
                        is_bundle: false
                        type: annotation
                        dir: '%kernel.project_dir%/src/Entity'
                        prefix: 'App\Entity'
                        alias: App
                    gedmo_translatable:
                        type: annotation
                        prefix: Gedmo\Translatable\Entity
                        dir: "%kernel.root_dir%/../vendor/gedmo/doctrine-extensions/lib/Gedmo/Translatable/Entity"
                        alias: GedmoTranslatable # (optional) it will default to the name set for the mapping
                        is_bundle: false
                    gedmo_translator:
                        type: annotation
                        prefix: Gedmo\Translator\Entity
                        dir: "%kernel.root_dir%/../vendor/gedmo/doctrine-extensions/lib/Gedmo/Translator/Entity"
                        alias: GedmoTranslator # (optional) it will default to the name set for the mapping
                        is_bundle: false
#                    gedmo_loggable:
#                        type: annotation
#                        prefix: Gedmo\Loggable\Entity
#                        dir: "%kernel.root_dir%/../vendor/gedmo/doctrine-extensions/lib/Gedmo/Loggable/Entity"
#                        alias: GedmoLoggable # (optional) it will default to the name set for the mappingmapping
#                        is_bundle: false
                    gedmo_tree:
                        type: annotation
                        prefix: Gedmo\Tree\Entity
                        dir: "%kernel.root_dir%/../vendor/gedmo/doctrine-extensions/lib/Gedmo/Tree/Entity"
                        alias: GedmoTree # (optional) it will default to the name set for the mapping
                        is_bundle: false

Also, in our stof_doctrine_extensions.yaml, let’s add the following configuration for the doctrine extensions.

#stof_doctrine_extensions.yaml
stof_doctrine_extensions:
    default_locale: en_US
    translation_fallback: true
    persist_default_translation: true
    orm:
        default:
            tree: true
            translatable: true
            sluggable: true

Now, if we try to update our schema

php bin/console doc:sch:update --dump-sql
The following SQL statements will be executed:

     CREATE TABLE ext_translations (id INT AUTO_INCREMENT NOT NULL, locale VARCHAR(8) NOT NULL, object_class VARCHAR(255) NOT NULL, field VARCHAR(32) NOT NULL, foreign_key VARCHAR(64) NOT NULL, content LONGTEXT DEFAULT NULL, INDEX translations_lookup_idx (locale, object_class, foreign_key), UNIQUE INDEX lookup_unique_idx (locale, object_class, field, foreign_key), PRIMARY KEY(id)) DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci ENGINE = InnoDB ROW_FORMAT = DYNAMIC;

Looks about right, so let’s hit it!

php bin/console doc:sch:update --force

And we get the following error

Updating database schema...


In AbstractMySQLDriver.php line 126:
                                                                                                                                
  An exception occurred while executing 'ALTER TABLE ext_translations CHANGE object_class object_class VARCHAR(255) NOT NULL':  
                                                                                                                                
  SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes               
                                                                                                                                

In PDOConnection.php line 109:
                                                                                                                   
  SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes  
                                                                                                                   

In PDOConnection.php line 107:
                                                                                                                   
  SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes

Now, after looking for this issue with Symfony and Gedmo Translations in the internet, I found this issue about Doctrine and this other one, and the fix here.

As davidbehler and javiereguiluz explain in the posts,

the cause of this is that Symfony advises you to use utf8mb4_general_ci/utf8mb4 as collation/charset for your database. utf8mb4 takes 4 bytes per char, meaning a 255 char field needs 1020 bytes for an index.

And so it seems that MySql 5.6 has a max key length of 767 bytes, so that leaves us with three options:

1. Decrease the field length to 191. However, we would have to override Gedmo’s translations bundle, as it uses 255 char fields everywhere…

2. We could change the charset encoding to ut8 like so (thanks to Kuba Florczuk for the configuration):

#doctrine.yaml
doctrine:
    dbal:
        # configure these for your database server
        driver: 'pdo_mysql'
        server_version: '5.6'
        charset: utf8
        default_table_options:
            charset: utf8
            collate: utf8_unicode_ci

        url: '%env(resolve:DATABASE_URL)%'
#...

3. We could upgrade to mysql 5.7 like so:

#docker-compose.yml
services:
    mysql:
      image: mysql:5.7
#...
#doctrine.yaml
doctrine:
    dbal:
        # configure these for your database server
        driver: 'pdo_mysql'
        server_version: '5.7'
        charset: utf8mb4
        default_table_options:
            charset: utf8mb4
            collate: utf8mb4_unicode_ci

        url: '%env(resolve:DATABASE_URL)%'
#...

All of them solutions work gracefully. Hope this helps, happy coding! 🙂

Gedmo’s Translations Documentation

StofDoctrineExensionBundle in Symfony’s official website.
Doctrine’s extensions GitHub doc.
Translatable’s GitHub doc.

Docker for Django

This blog post is a small guide for getting started with your Django environment with Docker. Since I got very positive feedback from the blog post about Docker and Symfony4, I decided to do the same with Docker and Django.

In this example we’re going to work with Django 1.11 (LTS), Python3 and MySql 5.6

Before we start, you’ll need to install Docker in your machine. You can download it from the official website.

Once Docker is installed, I strongly recommend playing with the getting started guide. Here the guide for macs and here the guide for windows. However, if you’re lazy like me, just use this command to make sure it’s installed.

docker --version

Once all of that is out of the way, we can start with the Django project and the Docker environment that will run it.

We have two cases I’m going to tap on.

Case 1: I’m creating a Django project from scratch and I want to set up a development environment with Docker.

Step1: Clone the docker-django repository which has the docker configuration files.

git clone https://github.com/joeymasip/docker-django.git

Step2: Create the Django project.

First off, let’s start docker containters with the project we just downloaded.

#cd to the location where you cloned the project
cd ~/Development/docker-django
#start the containers
docker-compose up -d 

This command starts the containers. The parameter -d makes them run in the background. If you omit the -d you’ll see the log.

Docker should start building (if it’s the first time for these images) and running with the containers in the background.

If you now try to run

docker ps

in your console, you’ll see that MySql container is running, but Django’s is not. This is normal, as we haven’t installed Django yet in our project.

Now we’ll create the django project with the following command (replace project_name for your project name)

docker-compose run django django-admin.py startproject project_name .

Note: Do not forget the . in the end

Step3: Create the Django application.

Now we’ll create the django application with the following command (replace app_name for your app’s name)

docker-compose run django python manage.py startapp app_name

Now, try running the same

docker-compose up -d

Now, if you run the docker ps

docker ps

This time, Django’s container will have been started.

So right now, if you just open your browser and type http://127.0.0.1:8000/ you should see it working, so you’re already set up to develop!

First off though, let’s update settings.py so we can use our mysql container instead of Django’s sqlite.

Step4: Update settings.

First let’s add your app_name in the installed apps

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    
    'app_name',
]

Now let’s configure your settings.py so the database settings points to the database service from docker.

Change:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    }
}

To:

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'docker_django_db',
        'USER': 'dbuser',
        'PASSWORD': 'dbpw',
        'HOST': 'mysql',
        'PORT': '3306',
        'TEST': {
            'NAME': 'docker_django_db_test',
        },
    }
}

Step5: Create the User model.

Under your app_name/models.py file, just create a User model that extends from Django’s Auth model.

#app_name/models.py
from django.db import models
from django.contrib.auth.models import AbstractUser


class User(AbstractUser):
    pass

Once this is done, just tell your project that we’ll be using our User model. This is best practices in case you ever need to make changes to the User model. So in your project_name settings.py, add this line:

AUTH_USER_MODEL = 'app_name.User'

Now our User is plugged in.

Step6: Run migrations.

To run migrations, we need to first enter the python django bash.

docker-compose exec django bash

Once in, we can make the migrations if it’s the first time we create the app, as we won’t have any.

python manage.py makemigrations

And also run them

python manage.py migrate

If you want to create an admin user to log in into Django’s admin panel,

python manage.py createsuperuser

That’s it!

Now just open your browser and type

http://127.0.0.1:8000/
http://127.0.0.1:8000/admin/

You’re already set up to develop, so happy coding with docker!

Case 2: I already have a Django project

Step1: Clone the docker-django repository which has the docker configuration files.

git clone https://github.com/joeymasip/docker-django.git

Step2: Move all files to your already created Django project.

1. The docker folder containing python + django and a MySQL container config for it.
2. The docker-compose.yml file
3. The .env file

Step3: Start the docker images inside your Django project folder.

cd into your Django project folder and type the following command

docker-compose up -d

This command starts the containers. The parameter -d makes them run in the background. If you omit the -d you’ll see the log.

Docker should start building (if it’s the first time for these images) and running with the containers in the background.

Step4: Update database settings.

So if you’re using the docker-compose.yml out of the box, your database name, user and pw need updating in your project’s settings.

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'docker_django_db',
        'USER': 'dbuser',
        'PASSWORD': 'dbpw',
        'HOST': 'mysql',
        'PORT': '3306',
        'TEST': {
            'NAME': 'docker_django_db_test',
        },
    }
}

Step5: Run migrations.

docker-compose exec django bash

Once in, we can run the migrations.

python manage.py migrate

Now, open a new chrome tab and type the following URL. The port is the one we set up in the docker-compose.yml

http://127.0.0.1:8000/
http://127.0.0.1:8000/admin/
http://127.0.0.1:8000/whatever-slug-you-want-from-your-project

You should see it working.

You’re already set up to develop, so happy coding with docker!

Docker for Symfony 4

This blog post is an introduction to devs who want to start using Docker with Symfony4. It will guide you through creating a Symfony 4 project running on Docker.

Before we start, you’ll need to install Docker in your machine. You can download it from the official website.

Once Docker is installed, I strongly recommend playing with the getting started guide. Here the guide for macs and here the guide for windows. However, if you’re lazy like me, just use this command to make sure it’s installed.

docker --version

Once all of that is out of the way, we can start with the Symfony 4 project and the Docker environment that will run it.

For the sake of the example, I’ve created a local environment running PHP5 to make things a bit trickier, since Symfony4 runs on PHP7.

Let’s try to create an empty SF4 skeleton on my local machine.

composer create-project symfony/skeleton symfony

Right off the bat I get the following error

[InvalidArgumentException]
Could not find package symfony/skeleton with stability stable in a version installable using your PHP version 5.6.29.

So as you can see, we’ve created an environment running PHP5 and we cannot create a SF4 project, which runs on PHP7. Docker should help us solve this problem 🙂

We have two cases I’m going to tap on.

Case 1: I’m creating a SF4 project from scratch and I want to set up a development environment with Docker.

Step1: Clone the docker-symfony4 repository which has the docker configuration files.

git clone https://github.com/joeymasip/docker-symfony4.git

Step2: Create the Symfony project skeleton.

First off, let’s start docker containters with the project we just downloaded.

#cd to the location where you cloned the project
cd ~/Development/docker-symfony4
#start the containers
docker-compose up -d 

This command starts the containers. The parameter -d makes them run in the background. If you omit the -d you’ll see the log.

Docker should start building (if it’s the first time for these images) and running with the containers in the background.

We will need to create the symfony project inside the php image bash, since it’s the only place we have PHP7. Remember we still have PHP5 in our machine, so first thing is to log into the bash for the php7 image.

docker-compose exec php-fpm bash

Once in there, we’re in a PHP7 image, so we should be able to create the skeleton for the symfony project.

#inside php-fpm bash
composer create-project symfony/skeleton symfony

Step3: Move the contents of the skeleton into the root of the application.

Unless you want to change the config of the working dir inside the docker-composer.yml, we need the symfony project to be in the root folder. Moreover, we can not clone the contents into the root folder directly like so (composer create-project symfony/skeleton .) because the installer deletes the contents of the folder you’re cloning into. Since it’s too risky, this option is not allowed. More info here.

Long story short, I’ve found this is the cleaner way to do it.

#inside php-fpm bash
mv /application/symfony/* /application
mv /application/symfony/.* /application

Now we can delete the empty folder we used for creating the skeleton

#inside php-fpm bash
rm -Rf /application/symfony

Step4: Require the components.

We can require whatever components we need.

#inside php-fpm bash
cd /application

composer require annotations
composer require --dev profiler
composer require twig
composer require orm
composer require form
composer require form validator
composer require maker-bundle

These are just a few, feel free to add the ones you want.

Step5: Creating some sample code in Symfony 4 project.

Now lets create a controller to test a sample route to make sure everything works.

#inside php-fpm bash
cd /application

bin/console make:controller Pizza

Now, open a new chrome tab and type the following URL. The port is the one we set up in the docker-compose.yml – If you check the dictionary of the ngix config, you can see that port 8000 maps the 80, which is the usual webserver port.

http://localhost:8000/pizza

We should now see our new controller action rendering a response.

Step6: Sync de database.

Finally, to sync the database, you need to update the .env file with the variables we set on the mysql image.

.env file that has been generated when requiring the orm package in Symfony4.

DATABASE_URL=mysql://db_user:db_password@127.0.0.1:3306/db_name

So if you check the docker-compose.yml file, you’ll see the credentials under the mysql configuration. So change the file to

DATABASE_URL=mysql://dbuser:dbpw@mysql:3306/docker_symfony4

Finally, let’s restart the containers

docker-compose down
docker-compose up -d
docker-compose exec php-fpm bash

Now inside the bash, you should be able to

#inside php-fpm bash
bin/console doc:sch:crea

You can connect an external database client such as Sequel Pro or MysqlWorkbench.

For the credentials remember to put again the ones we set in the mysql image.

Host: 0.0.0.0
Username: dbuser
Password: dbpw
Port: 8002

Also as a reminder, every time you composer down or kill the mysqld image, your schema will disapear! This not only means you’ll have to recreate again the following time, it also means the data will be lost. So make sure to dump the data if you need it later, or create some demo data/fixtures so you don’t have to add data manually.

Case 2: I already have a SF4 project

Maybe you cloned from elsewhere, or maybe you created it in the past with PHP7 in your local machine.

Step1: Clone the docker-symfony4 repository which has the docker configuration files.

git clone https://github.com/joeymasip/docker-symfony4.git

Step2: Move the files from the docker-symfony4 project folder you just cloned into your Symfony project root.

Move the docker-compose.yml and the folder named phpdocker containing nginx and php-fpm config for it to the root of your Symfony4 project.

Step3: Start the docker images inside your Symfony4 project folder.

cd into your Symfony project folder and type the following command

docker-compose up -d

This command starts the containers. The parameter -d makes them run in the background. If you omit the -d you’ll see the log.

Docker should start building (if it’s the first time for these images) and running with the containers in the background.

Now, open a new chrome tab and type the following URL. The port is the one we set up in the docker-compose.yml – If you check the dictionary of the ngix config, you can see that port 8000 maps the 80, which is the usual webserver port.

http://localhost:8000
http://localhost:8000/whatever-slug-you-want-from-your-project

You should see it working.

You’re already set up to develop, so happy coding with docker!

Disclaimer: The project container I created has been generated in phpdocker.io