brainlid.org

Pattern Matching to Format Text

Wed, 06 Mar 2019 05:00:00 -0700

Photo by Alex Block on Unsplash

Using string pattern matching to format text? Sure!

We store text like phone numbers, and Social Security Numbers (SSN) as unformatted text. When a user submits the data, we strip all formatting and non-number characters from it. If a user enters 800.555.4444, we store 8005554444. When it comes time to render the value into a PDF, document or export it in some other way, we format it in a consistent way like (800) 555-4444.

Elixir (and the BEAM) supports pattern matching binary strings, making this very clean and fun.

This is documented in Kernel.SpecialForms under «arg». You can use this in very powerful ways like matching on web protocols, parsing PNGs and more. However, sometimes the straight-forward “string” uses are harder to find documented. I’m hoping to add to that here.

The power is in the binary pattern match where I specify the length of each segment and bind it to a variable.

defmodule Utils.Format do
  def phone(<< three::binary-size(3), four::binary-size(4) >>) do
    "#{three}-#{four}"
  end
end

This example only matches a 7-character string.

Here’s the full version that also supports formatting a 10-character value where an area-code is included.

defmodule Utils.Format do
  @moduledoc """
  Functions for common string formatting.
  """

  @doc """
  Format a phone number with an expected area code.

  ## Examples

      iex> Format.phone("8005554444")
      "(800) 555-4444"

      iex> Format.phone("5554444")
      "555-4444"

      iex> Format.phone("short")
      "short"

      iex> Format.phone(nil)
      nil

  """
  @spec phone(value :: nil | String.t()) :: nil | String.t()
  def phone(value)
  def phone(nil), do: nil
  def phone(<< area::binary-size(3), three::binary-size(3), four::binary-size(4) >>) do
    "(#{area}) #{three}-#{four}"
  end
  def phone(<< three::binary-size(3), four::binary-size(4) >>) do
    "#{three}-#{four}"
  end
  def phone(other) when is_binary(other), do: other
end

Why not make this a library? I find that usually my application is pretty focused in what it needs. A public library needs to support many different formats and different use-cases to be valuable.

Elixir makes it so easy to do just what I need for my application that I don’t need to bring in a library or worry about creating one.

I love Elixir. I solved a need in my application without a single if conditional. It’s fully declarative, clear and performant.

Where can you use binary pattern matching?

Git Aliases

Tue, 13 Nov 2018 05:00:00 -0700

Photo by Dan Gold on Unsplash

Git is a version control tool used to track and manage source code files and changes to those files. In sharing some of my development tooling setup, I want to promote the use of git aliases as an “Express” route or shortcut to commonly used commands.

I don’t see enough people using them. People seem content to type out long commands that they use on a regular basis.

I just wanted to share a few of my favorite aliases here.

My Favorite Aliases

Git aliases are defined in the .gitconfig file in your Home directory.

[alias]
  b = branch
  r = remote -v
  s = status
  co = checkout
  uncommit = reset --soft HEAD^
  amend = commit --amend
  cp = cherry-pick -x

Example Usage

Instead of typing:

$ git checkout master

I can type:

$ git co master

Instead of:

$ git status

I only need to type:

$ git s

When I accidentally commit changes to the wrong branch, it’s so easy to just “uncommit” them.

$ git uncommit

Note: You should never uncommit something that has already been pushed to a remote.

What git aliases are your favorites?

People are Processes

Wed, 17 Jan 2018 10:00:00 -0700

Photo by rawpixel.com on Unsplash

When designing your Elixir systems, think about the processes as people. Now think of these people in an office all working together to serve the customer. That is how you can build your software systems in Elixir. It’s natural and intuitive!

A Person’s State

A person’s mental “state” is like an Elixir process’ state.

You know what you are thinking about.
You don’t know what someone else is thinking. It is private to them.
You can ask that person what they are thinking.
That person can choose to reply and tell you what they are thinking.

Elixir processes are very powerful. You might be tempted to think of a process as a “thread” in traditional “multi-threaded” programming but that’s wrong and will take you off course. “Threads” often share state and two people can’t do that. It’s unnatural and you end up resorting to things like mutexes to try and keep things from breaking.

Elixir’s processes provide a better way! And they are easier to think about too!

Your Software System is Like an Office

Your system is like an office. Let’s say the front door of your business is a website and a request just came in. Who services that request? Are multiple people involved in responding to that request?

If you have a Ruby on Rails or PHP system, then your business probably looks like a sole-proprietorship. A single-person operation. An individual request gets handled by a single process (or person). The business will do nothing else until that one request is dealt with.

We get around this limitation by cloning the whole business and hiding that behind a facade to make it look like one building. In tech terms, we run multiple instances of the whole system behind load balancers but each instance of the system can only handle one request at a time.

Elixir and Phoenix Scale Your Business!

When you build you business with Elixir and Phoenix, Phoenix takes care of hiring a new temp-worker for every request. That temp-worker handles the request and response. So, by just using Elixir and Phoenix, your system can handle more load with fewer resources and responds to each request faster.

That’s awesome! Your business serves more customers and adds employees on-demand and you don’t have to even think about it! You win!

Now, do you think it could get even better? Remember:

processes are like people
your system is like an office

Think of your system like an office. When you want to grow your business beyond the sole-proprietor model, how do you do that?

You hire people! You assign people roles or positions and have them work together to get more done! In fact, some of those people would be very specialized like the sales-gal and the finance-guy.

In tech terms, I create (hire) processes (people) and they send messages to each other to coordinate and now more people are working at the same time! This is how the real world works. This is how you work with other people. You understand this instinctively. The novelty is realizing that you can have a software system that works the same way.

Now that is a real business!

The sole-proprietor model is a bottle-neck. A small business will remain small and unable to service more customers until they figure out how to scale and hire more people. Don’t make that mistake with your software systems! Elixir has the power to help you build a smarter business. That means you have a smarter system.

You Can Do This! How?

Elixir can do this. You can do this with Elixir! What do you do?

Think of your system like an office
Think of processes as the people in the office
How would you solve the problem with real people?
Hire more people!

Use more processes! Concurrency for the win!

Want to Move Faster?

Contact me to help your team and project get to that point faster!

Want More Detail?

Want to go deeper and get a better understanding? I presented on this topic and more at an Elixir meetup. We make process and concurrency real!

Price's Law and Who's Doing the Work

Tue, 28 Nov 2017 11:20:45 -0700

Photo by jesse orrico on Unsplash

Price’s Law

The square root of the number of people in a domain do 50% of the work.

This means that in a company of 10 employees, 3 of them do 1/2 the work. The remaining 50% of the work is done by the other 7 people.

This scales too.

When there are 100 employees, 10 of them do 1/2 the work. The other 90% are doing the other half of the work.

As your company grows, incompetence grows exponentially and competence grows linearly.

Price’s Law specifically applies to creative work. Meaning the creation of something “new” like new products, ideas, software, music compositions, basketball shots made, artwork, etc.

The “Price” in Price’s Law is Derek J. de Solla Price.

Practical Impacts

I have seen this in my various workplaces. I also note when company leadership doesn’t even know who their top performers are. They don’t know that these 3-4 people are doing 50% of the important creative work of the company.

I have seen this law play out time and time again. I just didn’t know it was an actual law until recently. Now I am better able to pay attention and see it.

Knowing this now, it becomes incredibly important to identify who the top performs are in our organizations. They are worth keeping and add incredible value. They may be fairly quiet too.

I am a member of 3 different volunteer organizations, I work full time, and from time to time I have other creative efforts I works on outside of my other involvements.

I recognize that in 1 volunteer group, I am “along for the ride” and am coasting. While in another, I am very involved, creating and leading.

So the same person can occupy different creative positions and levels of involvement in their different social groups and at different times.

Short Video of Lecture

Jordan Peterson on Price’s Law - A portion of his lecture where he explains it well.

Notes from Lecture Video

10 employees, 3 of them do 1/2 the work
100 employees, 10 of them do 1/2 the work - that’s a problem, the other 90% are doing the other half.
10,000 employees 100 of them do 1/2 the work

As your company grows, incompetence grows exponentially and competence grows linearly.

When a company has poor performance over a couple quarters, the 100 talented people who have options, leave. They are doing half the work. Another round of layoffs comes, the next set of top performers leaves. The company is now in a death spiral.

This law applies in every single realm where there is creative production.

Applies to city size, baskets scored, hockey goals scored.

Compared to Monopoly game and the current 1% of rich people. It is an inevitable conclusion of iterated trading games. We don’t know how to fight it.

The top 1% end churns. It isn’t the same 1% all the time.

In companies, people get stuck in their niche and don’t move. The redistribution of people after losing the top performers doesn’t get picked back up very well by the remaining people. The effect is attenuated.

Companies can go into a death spiral that is almost impossible to get out of. Can happen extraordinarily quickly. Fortune 500 company’s churn. They only spend about 30 years at that level and value.

Creative people are troublesome to work with. How do you evaluate a creative person? They are continually creating new things that are difficult to evaluate.

Abusing Elixir Processes and State!

Sun, 24 Sep 2017 09:20:45 -0600

Recently I found a way of storing state in an Elixir Process that I hadn’t seen before. I never read about it in a book, never saw it in a talk and haven’t seen blog posts mentioning it. Did other people already know about this and it was frowned on? Did others just not know about it? But that couldn’t be, because I found it being used internally in the Elixir Logger module!

It’s called the “Process Dictionary”.

Before I dig in to the Dictionary and how easy it is to use and abuse, if you aren’t already familiar with the recommended ways to manage state with processes, check out my earlier post (or many others online).

I setup a simple Github project that makes it easy to play with Elixir processes and state management. https://github.com/brainlid/meetup_process_state

Logger’s Metadata

Logger has a powerful feature of tracking and including metadata on log entries. This is helpful for a Phoenix application when you want to tie all related log entries together using a unique request_id for the user’s request. In fact, Phoenix’s Plug.RequestId module does this.

To see it in action, here is a simple example. Note that this is using this Github project which configures Logger to output specified metadata.

    defmodule ProcessState.LoggerMeta do
      require Logger

      defp setup_metadata do
        Logger.metadata(state_1: "always_here")
      end

      defp step_1 do
        Logger.info("Step 1")
      end

      defp step_2 do
        Logger.info("Step 2")
      end

      defp step_3 do
        Logger.info("Step 3", [custom_1: "set_in_step_3"])
      end

      def run_steps do
        setup_metadata()
        step_1()
        step_2()
        step_3()
        {:ok, "Finished"}
      end
    end

Given the code above, an IEx terminal session might look like this…

    iex(1)> ProcessState.LoggerMeta.run_steps

    22:11:31.841 pid=<0.114.0> state_1=always_here [info]  Step 1

    22:11:31.841 pid=<0.114.0> state_1=always_here [info]  Step 2

    22:11:31.842 pid=<0.114.0> state_1=always_here custom_1=set_in_step_3 [info]  Step 3
    {:ok, "Finished"}

Did you notice that the “state_1=always_here” text in the log entry is on every line? That message was set once in the setup_metadata function with a call to Logger.metadata. Setting metadata on Logger this way is actually writing some state to the process running the code. It is showing up with every step. In step 3 some custom log metadata is written out as well. However, this will only show up for that one line.

How is this accomplished? The running process isn’t a GenServer or using tail-recursion to maintain state like the recommended approaches.

Looking at the Logger source, we find this…

    def metadata(keyword) do
      # [...]
      Process.put(@metadata, {enabled?, metadata})
      :ok
    end

Process Dictionary

An Elixir (and Erlang) process has a “Dictionary”. This is a local key-value pair storage area. Local here means it is internal to the process.

In Elixir, you can interface with it using these functions.

The only function not exposed through Elixir is the “erase/0” function. Which “erases the entire process dictionary. Returns the entire process diction before it was erased.” So perhaps we can see why that isn’t being made conveniently accessible.

However, if you need to call it, you can do so this way…

:erlang.erase()

Erlang Advanced Course has this to say about the Process Dictionary

Each process has a local store called the “Process Dictionary”.

[…]

Note that using the Process Dictionary:

Destroys referencial transparency

Makes debugging difficult

Survives Catch/Throw

So:

Use with care

Do not over use - try the clean version first

Now that we’ve introduced the Process Dictionary, lets abuse it!

defmodule ProcessState.DictionaryAbuse do
  require Logger

  def state_1 do
    Process.put(:state, :greeting)
    :ok
  end

  def state_2 do
    Process.put(:state, :count_to_10)
    :ok
  end

  def state_3 do
    Process.put(:state, :random)
    :ok
  end

  def run do
    case Process.get(:state) do
      :greeting ->
        "Hello there!"

      :count_to_10 ->
        Enum.each(1..10, fn(num) ->
          IO.puts("#{inspect num}...")
          Process.sleep(500)
        end)

      :random ->
        sayings = ["Well, that's just great.", "Sorry?", "Eh?", "Go on then!"]
        Enum.random(sayings)

      _ ->
        "Try again!"
    end
  end
end

In IEx we can interact with it like this…

      alias ProcessState.DictionaryAbuse
      DictionaryAbuse.state_1
      DictionaryAbuse.run

      DictionaryAbuse.state_2
      DictionaryAbuse.run

      DictionaryAbuse.state_3
      DictionaryAbuse.run

      Process.put(:state, "other")
      DictionaryAbuse.run

We might see something like this…

    iex(1)> alias ProcessState.DictionaryAbuse
    ProcessState.DictionaryAbuse
    iex(2)> DictionaryAbuse.state_1
    :ok
    iex(3)> DictionaryAbuse.run
    "Hello there!"

    iex(4)> DictionaryAbuse.state_2
    :ok
    iex(5)> DictionaryAbuse.run
    1...
    2...
    3...
    4...
    5...
    6...
    7...
    8...
    9...
    10...
    :ok

    iex(6)> DictionaryAbuse.state_3
    :ok
    iex(7)> DictionaryAbuse.run
    "Go on then!"
    iex(8)> DictionaryAbuse.run
    "Sorry?"
    iex(9)> DictionaryAbuse.run
    "Eh?"

Now imagine writing a unit test for the “run/0” function. It isn’t a “pure” function. It isn’t obviously predictable. Imagine running across similar code in a project. It takes more effort to understand it.

What may not be immediately apparent above is that we are modifying the state of the IEx process running our commands.

So we can use Process.get/1 to examine the current :state value we set in the state_x functions from IEx and it can access the state.

    iex(10)> Process.get(:state)
    :random

Lets try changing it to something that isn’t supported by the code in the module.

    iex(11)> Process.put(:state, "other")
    :random
    iex(12)> DictionaryAbuse.run
    "Try again!"

Using Process.get/0, we can get the whole Process Dictionary returned to see what it looks like.

    iex(13)> Process.get()
    [iex_history: %IEx.History.State{queue: {[{12, 'Process.get(:state)\n',
         :random}, {11, 'DictionaryAbuse.run\n', "Go on then!"},
        {10, 'DictionaryAbuse.run\n', "Eh?"},
        {9, 'DictionaryAbuse.run\n', "Eh?"},
        {8, 'DictionaryAbuse.state_3\n', :ok}, {7, '\n', nil},
        {6, 'DictionaryAbuse.run\n', :ok},
        {5, 'DictionaryAbuse.state_2\n', :ok}, {4, '\n', nil},
        {3, 'DictionaryAbuse.run\n', "Hello there!"},
        {2, 'DictionaryAbuse.state_1\n', :ok}],
       [{1, 'alias ProcessState.DictionaryAbuse\n',
         ProcessState.DictionaryAbuse}]}, size: 12, start: 1},
     # [...],
     state: :random,
     "$ancestors": [#PID<0.58.0>],
     "$initial_call": {IEx.Evaluator, :init, 4}]

What’s this? The IEx.Evaluator module that runs our commands in out beloved IEx is storing the session’s command history in the Process Dictionary! It’s even storing the results of the functions.

Also we see that “$ancestors” and “$initial_call” are stored in every process. The $ancestors is the owning process and $initial_call is what was used to start this process.

In fact, we can see the Process Dictionary in Observer when we view the IEx process. There is a “Dictionary” tab.

    # which process is the IEx.Evaluator?
    iex> self()
    #PID<0.114.0>

    iex> :observer.start

Curious! The Observer Dictionary page doesn’t include the :iex_history that we see when calling Process.get/0. Mysteries remain!

However, it’s time close out this long post looking at the Elixir/Erlang Process Dictionary.

Conclusion

Logger and IEx both use the Process Dictionary. They both seem to be good and appropriate uses. We should be aware of this tool so we can call on it when it is appropriate. I hope it is clear though that it can be easily abused making code harder to test and reason about. If you abuse it, be prepared to take some heat from your team.

The goal here is to become aware of another, less talked about way of storing state in a Process.

For the final words, I’ll share what Fred Hebert, the author of Learn You Some Erlang for Great Good and Erlang in Anger, had this to say on the topic. [source]

The process dictionary is not inherently evil, but it has many drawbacks that have been mentioned countless times already: it is harder to debug and reason about, it has different semantics than most of the language, it breaks when you try to send state to other processes without knowing it’s tied to the process’ very existence, it is not garbage collected until the process dies, it is hard to replace by a different key-value store and it angers people (I do consider this to be a negative). On the positive side, you have better update speed. On the neutral side, you have global access to some values, within the scope of the process: this can both be useful (static config) or dangerous (global scope!)

Look for the tradeoffs you’re ready to make, what your application actually needs. Use the right tool for the right job and make sure the process-global aspect of it and the speed do warrant all the downsides of it. I hope this helps.

Elixir Processes and State - Recommended

Fri, 22 Sep 2017 20:20:45 -0600

Taking the “happy path”

In Elixir and Erlang, a process is where we keep our state.

The story goes like this:

a process starts with an initial state
the process waits for messages
send messages to the process to change it’s state
process modifies it’s state and passes itself the new version of it’s state
waits for messages (rinse and repeat)

That’s it.

There are a number of different interfaces to do this, but underneath it all, they are all doing the same thing.

The built-in ways look like this:

spawn a process
Use a GenServer behavior
Use an Agent behavior (a subset of GenServer functionality)

Why is this a recommended pattern?

it is efficient
it is easy to write pure functions that test the behavior (given the same input, the same output results)
it is common convention, others will understand what you are doing

Examples of Recommended Style

I setup a simple Github project that makes it easy to play with these approaches.

https://github.com/brainlid/meetup_process_state

Basic Spawn Example

    defmodule ProcessState.SpawnBasic do
      require Logger

      def start(initial_state) do
        Logger.info("Starting separate process with initial state: #{inspect initial_state}")
        spawn __MODULE__, :main_loop, [initial_state]
      end

      def main_loop(state) do
        Logger.info("main_loop state: #{inspect state}")
        new_state =
          receive do
            {:add, increment} ->
              Logger.info("Received :add message: #{inspect increment}")
              add(state, increment)
            _ ->
              Logger.info("Received some other message. State unchanged.")
              state
          end
        main_loop(new_state)
      end

      defp add(%{counter: current} = _state, increment) do
        %{counter: current + increment}
      end
    end

When we call start, it spawns a new process. That process sets up it’s initial state and calls main_loop. There is nothing special about this function name. I just picked something. The process then calls receive. This blocks the process until it receives a message. This example only responds to a message of {:add, number}. When that message is received, it modifies the state and uses “tail recursion” to call itself again with the new state.

Experiment by sending messages and seeing the log output in the console. The Logger config was customized to output the pid as part of the metadata. This makes it easier to see which process is running which commands.

You might see something like this…

    iex(1)> pid = ProcessState.SpawnBasic.start(%{counter: 0})

    19:47:08.640 pid=<0.114.0> [info]  Starting separate process with initial state: %{counter: 0}

    19:47:08.640 pid=<0.116.0> [info]  main_loop state: %{counter: 0}
    #PID<0.116.0>

    iex(2)> send(pid, {:add, 10})

    19:47:12.469 pid=<0.116.0> [info]  Received :add message: 10

    19:47:12.469 pid=<0.116.0> [info]  main_loop state: %{counter: 10}
    {:add, 10}

    iex(3)> send(pid, {:add, 1})

    19:47:14.758 pid=<0.116.0> [info]  Received :add message: 1
    {:add, 1}

    19:47:14.758 pid=<0.116.0> [info]  main_loop state: %{counter: 11}

If you open Observer and inspect the process (in this example it would be 0.116.0) the “State” tab will show you “Information could not be retrieved, system messages may not be handled by this process.” This basic spawn works as intended, but it doesn’t support being inspected and handling system-level messages. That’s one reason it’s better to use a GenServer.

GenServer Example

The GenServer approach is preferred. It makes it easier to supervise the process and it has built-in support for handling system messages.

    defmodule ProcessState.GenServerState do
      require Logger
      use GenServer

      # Client

      def start(initial_state) do
        Logger.info("Starting genserver with initial state: #{inspect initial_state}")
        GenServer.start(__MODULE__, initial_state)
      end

      def add(pid, increment) do
        Logger.info("Sending :add message")
        GenServer.call(pid, {:add, increment})
      end

      # Server (callbacks)

      def handle_call({:add, increment}, _from, %{counter: current} = _state) do
        Logger.info("Adding #{inspect increment} to state.")
        new_state = %{counter: current + increment}
        Logger.info("New state is #{inspect new_state}")
        {:reply, current + increment, new_state}
      end

    end

Here’s an example of how to interact with it in IEx.

    alias ProcessState.GenServerState
    {:ok, pid} = GenServerState.start(%{counter: 0})
    GenServerState.add(pid, 10)
    GenServerState.add(pid, 11)
    :observer.start
    # examine the "state" of the process

You might see something like this…

    iex(1)> alias ProcessState.GenServerState
    ProcessState.GenServerState

    iex(2)> {:ok, pid} = GenServerState.start(%{counter: 0})

    19:59:29.725 pid=<0.114.0> [info]  Starting genserver with initial state: %{counter: 0}
    {:ok, #PID<0.117.0>}

    iex(3)> GenServerState.add(pid, 10)
    10

    19:59:32.965 pid=<0.114.0> [info]  Sending :add message

    19:59:32.965 pid=<0.117.0> [info]  Adding 10 to state.

    19:59:32.965 pid=<0.117.0> [info]  New state is %{counter: 10}

    iex(4)> GenServerState.add(pid, 11)
    21

    19:59:38.822 pid=<0.114.0> [info]  Sending :add message

    19:59:38.822 pid=<0.117.0> [info]  Adding 11 to state.

    19:59:38.822 pid=<0.117.0> [info]  New state is %{counter: 21}

Note the pid responsible for running each line? All the same things are happening here that are happening in the spawn example. The GenServer approach makes it less explicit and adds support for handling other system messages.

Running Observer, I found the pid (0.117.0 in my case) in the “Processes” tab. Inspecting that process, the “State” tab looks like this.

The last line shows the “state” structure in the GenServer process.

Conclusion

Using the GenServer or Agent approach for storing runtime state in your Elixir application is the standard, recommended way. They have built-in support for being supervised. There is built-in support for handling system-level events. Observer understands them and can work with them natively. Most importantly perhaps, the functions can be “pure” and predictable. Given the same input, the same output is returned.

This is the “clean” version of managing state in a process.

Security Scanning Your Phoenix App

Wed, 14 Jun 2017 22:20:45 -0600

Sobelow is a security-focused static analysis tool for the Phoenix framework. It is produced by NCC Group, a cyber-security firm whose roots go back to June 1999.

At the time of this writing, the current version is 0.3.6 and it has been gaining traction in the Elixir/Phoenix communities.

Why Run Security Scans?

First, it should be a matter of personal responsibility to your product and your users that you try to protect your application and your user’s data.

Second, various certifications like PCI Compliance can require things like automated static-analysis security scans or security code reviews of all changes to the system.

Third, what coding habits do you have that might be a security risk? Sobelow can be that third-party reviewer to challenge your long-held assumptions.

I work at a fast-paced FinTech startup with agile workflows. The best option is to have security scans automatically performed on PRs before they are merged. Some benefits to having them automated is you don’t forget to run the checks and it helps identify new code changes that could have security problems.

Install Options

The project recommends installing it outside of the project to be scanned like this.

$ mix archive.install hex sobelow

This works well when you want to scan other people’s projects without modifying their dependencies.

It also works to add it to your project’s mix.exs if that makes more sense for your situation.

def deps do
  [{:sobelow, "~> 0.3", only: [:dev]}]
end

Running a Scan

Running your first scan is easy.

$ mix sobelow

There are a number of command-line options and new options were recently added so it is worth checking for current documentation.

We Need Code to Scan!

I did a quick Github search for OpenSource Phoenix applications that I could use for this post and found one under active development. This will work perfectly!

Setup steps before we can test sobelow on the project.

Clone the project locally
Make it able to compile
Nothing more needed! It’s “static-analysis” remember?

Run sobelow on the project…

    $ mix sobelow

    ##############################################
    #                                            #
    #          Running Sobelow - v0.3.6          #
    #  Created by Griffin Byatt - @griffinbyatt  #
    #     NCC Group - https://nccgroup.trust     #
    #                                            #
    ##############################################

    Missing CSRF Protections - High Confidence
    Pipeline: browser:4

    -----------------------------------------------

    HTTPS Not Enabled - High Confidence

    -----------------------------------------------

    Known Vulnerable Dependency - Plug v1.3.4

    -----------------------------------------------

    Known Vulnerable Dependency - Phoenix v1.2.1

    -----------------------------------------------

    Directory Traversal in `File.read!` - High Confidence
    File: web/controllers/contact_controller.ex - create_nested_data:175
    Variable: temp_file

    -----------------------------------------------

    Directory Traversal in `File.rm!` - High Confidence
    File: web/controllers/contact_controller.ex - create_nested_data:175
    Variable: temp_file

    -----------------------------------------------

    Directory Traversal in `File.read!` - High Confidence
    File: web/controllers/contact_controller.ex - view_uploaded_data:159
    Variable: temp_file

    -----------------------------------------------

    Directory Traversal in `File.read!` - Medium Confidence
    File: web/controllers/contact_controller.ex - import_data:135
    Variable: temp_file

    -----------------------------------------------

    Directory Traversal in `File.rm!` - Medium Confidence
    File: web/controllers/contact_controller.ex - import_data:135
    Variable: temp_file

    -----------------------------------------------

    Insecure use of `File` and `Path` - Low Confidence
    File: web/controllers/contact_controller.ex - import_data:135
    Variable: upload

    -----------------------------------------------

    ... SCAN COMPLETE ...

Note: Unfortunately, the color coding doesn’t translate to my copy of the text here. So just note that it is sorted with “High Confidence” items higher up then descending to “Medium” and “Low”. The colors change accordingly.

Interesting output! Now what does it mean? It is worth noting what the Sobelow project says about “False Positives”:

Sobelow favors over-reporting versus under-reporting. As such, you may find a number of false positives in a typical scan. These findings may be individually ignored by adding a @sobelow_skip mark, along with a list of modules, before the function definition.

So we need to check them personally and determine if it is a problem or not.

Let’s try out the command-line option --with-code and see if that helps give context to the warnings. I won’t include the full output for brevity’s sake.

    Missing CSRF Protections - High Confidence
    Pipeline: browser:4


    pipeline(:browser) do
      plug(:accepts, ["html"])
      plug(:fetch_session)
      plug(:fetch_flash)
      plug(:put_secure_browser_headers)
    end

I double-checked the router file and yes, it is missing the plug :protect_from_forgery call you would otherwise expect.

    Known Vulnerable Dependency - Plug v1.3.4
    Details: Header Injection
    CVE: TBA

    -----------------------------------------------

    Known Vulnerable Dependency - Phoenix v1.2.1
    Details: Arbitrary URL Redirect
    CVE: TBA

Well that’s nice! A recent sobelow feature is checking for known vulnerabilities in the project’s dependencies. And yes, the project’s mix.exs file is pinned to an older phoenix version. Time to upgrade!

{:phoenix, "~> 1.2.1"}

Here’s another one with a good code sample.

    Directory Traversal in `File.read!` - High Confidence
    File: web/controllers/contact_controller.ex - view_uploaded_data:159
    Variable: temp_file

    def(view_uploaded_data(conn, %{"mapping" => mapping, "temp_file" => temp_file})) do
      table = File.read!("tmp/#{temp_file}.csv") |> ExCsv.parse!() |> ExCsv.with_headings() |> Enum.to_list()
      total_rows = Enum.count(table) - 1
      contact_headers = Map.keys(mapping["contact"])
      organization_headers = Map.keys(mapping["organization"])
      first_row = Enum.at(table, 0)
      contact_values = for({db_col, csv_col} <- mapping["contact"]) do
        first_row[csv_col]
      end
      organization_values = for({db_col, csv_col} <- mapping["organization"]) do
        first_row[csv_col]
      end
      json(conn, %{contact_headers: contact_headers, organization_headers: organization_headers, contact_values: contact_values, organization_values: organization_values, temp_file: temp_file})
    end

Again, the coloring doesn’t come across here. But the code being pointed out is this…

File.read!("tmp/#{temp_file}.csv")

After checking the router, it is clear that this is indeed taking user submitted data and interpolating that text into a command-line path. This means submitting a “temp_file” value like “../somewhere/else” poses a risk of accessing data from anywhere on disk. I verified in the router that in order to even make this call, you must be an authenticated user. This prompts the question, “How much do you trust all of your users?”

I notified the project on Github of the issues and they have already addressed them.

It is also interesting how sobelow displays the function as def(view_uploaded_data(...)). Looking at the actual file, it was as you’d expect with def view_uploaded_data(...). I guess that’s exposing the reality that def is an Elixir macro. :)

Now you’ve seen some real-world examples of how sobelow can help your project.

Umbrella Projects

Sobelow is only designed for scanning Phoenix applications. So it doesn’t scan a standard mix new project. I follow the pattern where my projects are umbrella applications with the Phoenix project being generated without Ecto. This really reduces the utility of sobelow.

Running it from the root of my project it returns this…

$ mix sobelow
This does not appear to be a Phoenix application.

Luckily, the --root option lets me specify the path to the Phoenix app in my project.

    $ mix sobelow --root apps/my_web/

    ##############################################
    #                                            #
    #          Running Sobelow - v0.3.6          #
    #  Created by Griffin Byatt - @griffinbyatt  #
    #     NCC Group - https://nccgroup.trust     #
    #                                            #
    ##############################################

    HTTPS Not Enabled - High Confidence

    -----------------------------------------------

    ... SCAN COMPLETE ...

It’s totally true. My personal project that lives only on my machine hasn’t been setup for HTTPS yet. But, it’s a nice reminder that this should be a priority before I deploy outside of my home.

But it found nothing else. Since all my Ecto queries are done in other apps where the business logic lives, how comfortable should I be that this didn’t find anything? I’m not sure. Sobelow has been advancing quickly recently. I’m hopeful that it will grow in sophistication over time.

Work Project Scan Results

I started using Sobelow on our backend system at work. Now, I got the same “HTTPS Not Enabled”. However, in this case it isn’t a problem. Our production deployment has the web application behind load-balancers that are terminating the HTTPS connection. So our app doesn’t have that responsibility. How do we tell Sobelow this is okay?

Disable a Check

To disable the “HTTPS Not Enabled” check, I searched the sobelow project code for some error text and found it easily enough.

This one is not a specific line of code to disable, it is turning off the check. The -i is to “ignore” that check.

$ mix sobelow -i Config.HTTPS

Valid Finds

Sobelow found a valid “Unsafe String.to_atom - High Confidence” issue. At work, many of the backend developers are new to Elixir. You can’t expect everyone to be familiar with the details of a new language and its runtime. A member of our team took client provided raw param data and performed String.to_atom on it. Atoms in Elixir are like Symbols in Ruby. Elixir atoms are not garbage collected. This allows a malicious user to arbitrarily create them through an HTTP POST which can be abused to consume system resources until the server crashes. Resulting in a “denial of service” attack.

This find lets us have an “education moment”. So it works!

Automated Scanning

The option --exit returns a non-zero return code when problems are found. This makes it easy for an automated system to “fail” the process when a new vulnerability is introduced.

From the docs:

The exit option accepts a confidence threshold (low, medium, or high), and will return a non-zero exit status at or above that threshold.

$ mix sobelow --exit Low

So our project at work can run the tests on the multiple phoenix applications in the umbrella and automatically fail the code checks when new problems are introduced.

Conclusion

Sobelow is the first static-analysis security scanner for Phoenix that I’m aware of. I looked around a lot. At work, we have PCI Compliance requirements in addition to our desire to have a completely secure system. Sobelow came along at the perfect time for us. It has become part of our automated processes for testing code quality.

Give it a try on your Phoenix projects and share your results!

History in IEx between sessions

Wed, 22 Feb 2017 04:45:45 -0700

When working with Elixir in an IEx console session, sometimes it is painful that it forgets everything between sessions!

There is a simple Github project that fixes this!

git clone https://github.com/ferd/erlang-history.git
cd erlang-history
sudo make install

You can read more about it:

This project patches your local Erlang install to add history support. You will have to re-run this process after updating your Erlang version.

I cannot personally guarantee the code. Since it clones the repo each time it is run, you should check for any changes since the last time you audited it.

Atom Editor for Elixir Development (Revisited)

Mon, 20 Feb 2017 05:30:45 -0700

A lot has changed in the Elixir community since I wrote my previous post on this topic.

It’s time to update what Atom plugins work well for Elixir development.

Atom’s Elixir Development Plugins

The number of needed plugins has gone down. That’s a good thing! It means that the quality of the options is higher. I’m breaking this into 2 categories.

Must have
Optional

NOTE: Install these plugins through Atom’s Settings > Install. See the Atom Editor website for getting started with Atom.

Must Have Elixir Plugins

language-elixir - Syntax support. Really, you have to have this.
atom-elixir - Advanced code completion and much, much more.

The new one here is atom-elixir. It wasn’t available the last time I reviewed the options. It does such a good job that it replaces multiple other plugins.

Here’s a little on how it works. There is a project called Alchemist that was created for supporting Elixir in Emacs. A spin-off project (by the same author) called alchemist-server aims to make that same Elixir information available in an editor-independent way. The atom-elixir plugin leverages alchemist-server to do all the hard work on getting Elixir intelligence and makes it accessible in Atom.

This is meaningful to me because it indicates that support and features are more likely to continue.

I want to draw special attention to the atom-elixir plugin because it still has relatively low download numbers but is my personal go-to tool for Elixir development. I believe more people need to know about it. And no, I have nothing to do with the project or the people involved. :)

Take a Look…

Screenshot of the auto-completion window:

Screenshot of the documentation tab that opens on demand:

Check the shortcuts for your platform, but with a cursor on a function, I press atl-down (arrow) and it performs a “goto-definition”. I can then jump again and again going deeper. Then, using alt-up, I can pop the stack and jump back to where my cursor was at the previous point. This jumping back keeps popping the stack to eventually return me to the first jump point. I really love this feature!

Caveat

Atom allows you to open multiple projects at the same time. They show up in the tree with different roots. However, the code-completion doesn’t work with multiple projects open like that. It runs 1 alchemist server and it can’t cover both projects at the same time. However, it works fine with an Umbrella project.

Optional Elixir Plugins

linter-elixirc - Displays in-line the error with parsed out error message text. Runs after saving the file. Works by compiling the project and displaying the first error it encounters. Works well. In the past is has reported some false positives for me but it appears to be doing better as of late.
linter - a base linter provider that other language specific linters build on. Used by “linter-elixirc”.

Conclusion

It’s been fun working in the Elixir ecosystem and community. Lots of great advancements have been made in a relatively short time. I can’t wait to see how things improve and change going forward!

Here I highlighted a small number of Atom plugins that help with developing Elixir applications. If you aren’t already using atom-elixir, you should try it out now.

Want to share your favorite Elixir-related Atom plugin? Share it here!

Atom and CTAGS not in PATH Error

Mon, 20 Feb 2017 04:30:45 -0700

I use the Atom Editor on Linux. Arch Linux specifically. Atom has a nice feature of ctrl-R to view the symbols in the current file. This makes it easy to jump to a function or declaration in the current file.

For the past while, it’s been broken for me. I decided it was time to dig in and solve it.

I learned that ctags is now required to be provided by the system. This blog post helped identify that and offers some great tips! It references this SO answer too.

For my Arch Linux install, I installed ctags.

$ sudo pacman -S ctags

Yay! Now running ctrl-R on a file parses and shows the symbols/functions in the file. This was particularly a problem for me with Elixir source files.

However, the ctrl-shift-R for project-wide searching comes up empty.

The same blog post shows how to have the ctags rebuilt for the project on checkout of a different branch. (Using githooks).

Mystery solved. Phew!

brainlid.org

Pattern Matching to Format Text

Git Aliases

My Favorite Aliases

Example Usage

People are Processes

A Person’s State

Your Software System is Like an Office

Elixir and Phoenix Scale Your Business!

You Can Do This! How?

Want to Move Faster?

Want More Detail?

Price's Law and Who's Doing the Work

Price’s Law

Practical Impacts

Short Video of Lecture

Notes from Lecture Video

Abusing Elixir Processes and State!

Logger’s Metadata

Process Dictionary

Conclusion

Elixir Processes and State - Recommended

Examples of Recommended Style

Basic Spawn Example

GenServer Example

Conclusion

Security Scanning Your Phoenix App

Why Run Security Scans?

Install Options

Running a Scan

We Need Code to Scan!

Umbrella Projects

Work Project Scan Results

Disable a Check

Valid Finds

Automated Scanning

Conclusion

History in IEx between sessions

Atom Editor for Elixir Development (Revisited)

Atom’s Elixir Development Plugins

Must Have Elixir Plugins

Take a Look…

Code Navigation Feature

Caveat

Optional Elixir Plugins

Conclusion

Atom and CTAGS not in PATH Error