Jekyll can be confusing at first. Jekyll actually only does one of a few things to your files, although it classifies files in many different ways.

Jekyll either copies, omits or transforms files from your source directory into an output directory.1 When Jekyll transforms your file, it will always be in the following way, potentially skipping steps.2

If a file begins with yaml frontmatter, Jekyll will apply the following transformation to that file:

  1. liquid templating: liquid template parsing is run on the original content of the file, with several variables, like site or page provided as arguments your liquid templates can access.
  2. converting the content: Jekyll asks converters whether they match your file’s extension, e.g. kramdown matches .md, and if so they run against the output from template parsing. Different converters run depending on your file’s extension, e.g. .md extension and the kramdown converter.
  3. layout parsing: The output of conversion is passed as the {{content}} variable to either your default layout, or the layout specified in your file’s yaml frontmatter.
  4. The output of the layout is written to a file in your output directory, or omitted.

Now I want to walk through an example of Jekyll applying this transformation. Later in Jekyll’s core and Testing a complete build we will look at the general structure Jekyll’s algorithm follows and what sorts of files get which treatment.

Jekyll’s transformation

We can find Jekyll’s transformation in Renderer.run, which essentially does the following, potentially skipping some of the steps:

after_liquid = render_with_liquid(file_content) # line 62
after_markdown = convert(after_liquid) # line 66
place_in_layout(after_markdown) # line 71

So if we’re rendering this post:

 ---
 layout: post
 title:  "Welcome to Jekyll!"
 date:   2016-08-17 23:50:36 -0400
 categories: jekyll update
 ---
 You’ll find this post in your `_posts` directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run `jekyll serve`, which launches a web server and auto-regenerates your site when a file is updated.

 To add new posts, simply add a file in the `_posts` directory that follows the convention `YYYY-MM-DD-name-of-post.ext` and includes the necessary front matter. Take a look at the source for this post to get an idea about how it works.

 Jekyll also offers powerful support for code snippets:

 {% highlight ruby %}
 def print_hi(name)
   puts "Hi, #{name}"
 end
 print_hi('Tom')
 #=> prints 'Hi, Tom' to STDOUT.
 {% endhighlight %}

 Check out the [Jekyll docs][jekyll-docs] for more info on how to get the most out of Jekyll. File all bugs/feature requests at [Jekyll’s GitHub repo][jekyll-gh]. If you have questions, you can ask them on [Jekyll Talk][jekyll-talk].

 [jekyll-docs]: http://jekyllrb.com/docs/home
 [jekyll-gh]:   https://github.com/jekyll/jekyll
 [jekyll-talk]: https://talk.jekyllrb.com/{% raw %}

… you can see the output from the first two stages of transformation:

input
1. liquified
2. converted
You’ll find this post in your `_posts` directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run `jekyll serve`, which launches a web server and auto-regenerates your site when a file is updated.

To add new posts, simply add a file in the `_posts` directory that follows the convention `YYYY-MM-DD-name-of-post.ext` and includes the necessary front matter. Take a look at the source for this post to get an idea about how it works.

Jekyll also offers powerful support for code snippets:

{% highlight ruby %}
def print_hi(name)
  puts \"Hi, \#{name}\"
end
print_hi('Tom')
#=> prints 'Hi, Tom' to STDOUT.
{% endhighlight %}

Check out the [Jekyll docs][jekyll-docs] for more info on how to get the most out of Jekyll. File all bugs/feature requests at [Jekyll’s GitHub repo][jekyll-gh]. If you have questions, you can ask them on [Jekyll Talk][jekyll-talk].

[jekyll-docs]: http://jekyllrb.com/docs/home
[jekyll-gh]:   https://github.com/jekyll/jekyll
[jekyll-talk]: https://talk.jekyllrb.com/
You’ll find this post in your `_posts` directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run `jekyll serve`, which launches a web server and auto-regenerates your site when a file is updated.

To add new posts, simply add a file in the `_posts` directory that follows the convention `YYYY-MM-DD-name-of-post.ext` and includes the necessary front matter. Take a look at the source for this post to get an idea about how it works.

Jekyll also offers powerful support for code snippets:


<figure class=\"highlight\"><pre><code class=\"language-ruby\" data-lang=\"ruby\"><span class=\"k\">def</span> <span class=\"nf\">print_hi</span><span class=\"p\">(</span><span class=\"nb\">name</span><span class=\"p\">)</span>
  <span class=\"nb\">puts</span> <span class=\"s2\">\"Hi, </span><span class=\"si\">\#{</span><span class=\"nb\">name</span><span class=\"si\">}</span><span class=\"s2\">\"</span>
<span class=\"k\">end</span>
<span class=\"n\">print_hi</span><span class=\"p\">(</span><span class=\"s1\">'Tom'</span><span class=\"p\">)</span>
<span class=\"c1\">#=&gt; prints 'Hi, Tom' to STDOUT.</span></code></pre></figure>


Check out the [Jekyll docs][jekyll-docs] for more info on how to get the most out of Jekyll. File all bugs/feature requests at [Jekyll’s GitHub repo][jekyll-gh]. If you have questions, you can ask them on [Jekyll Talk][jekyll-talk].

[jekyll-docs]: http://jekyllrb.com/docs/home
[jekyll-gh]:   https://github.com/jekyll/jekyll
[jekyll-talk]: https://talk.jekyllrb.com/
<p>You’ll find this post in your <code class=\"highlighter-rouge\">_posts</code> directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run <code class=\"highlighter-rouge\">jekyll serve</code>, which launches a web server and auto-regenerates your site when a file is updated.</p>

<p>To add new posts, simply add a file in the <code class=\"highlighter-rouge\">_posts</code> directory that follows the convention <code class=\"highlighter-rouge\">YYYY-MM-DD-name-of-post.ext</code> and includes the necessary front matter. Take a look at the source for this post to get an idea about how it works.</p>

<p>Jekyll also offers powerful support for code snippets:</p>

<figure class=\"highlight\"><pre><code class=\"language-ruby\" data-lang=\"ruby\"><span class=\"k\">def</span> <span class=\"nf\">print_hi</span><span class=\"p\">(</span><span class=\"nb\">name</span><span class=\"p\">)</span>
  <span class=\"nb\">puts</span> <span class=\"s2\">\"Hi, </span><span class=\"si\">\#{</span><span class=\"nb\">name</span><span class=\"si\">}</span><span class=\"s2\">\"</span>
<span class=\"k\">end</span>
<span class=\"n\">print_hi</span><span class=\"p\">(</span><span class=\"s1\">'Tom'</span><span class=\"p\">)</span>
<span class=\"c1\">#=&gt; prints 'Hi, Tom' to STDOUT.</span></code></pre></figure>

<p>Check out the <a href=\"http://jekyllrb.com/docs/home\">Jekyll docs</a> for more info on how to get the most out of Jekyll. File all bugs/feature requests at <a href=\"https://github.com/jekyll/jekyll\">Jekyll’s GitHub repo</a>. If you have questions, you can ask them on <a href=\"https://talk.jekyllrb.com/\">Jekyll Talk</a>.</p>

Then in the last stage we put it into our template’s {{content}}:

template body
3. final output
<article class=\"post\" itemscope itemtype=\"http://schema.org/BlogPosting\">

  <header class=\"post-header\">
    <h1 class=\"post-title\" itemprop=\"name headline\">{{ page.title | escape }}</h1>
    <p class=\"post-meta\"><time datetime=\"{{ page.date | date_to_xmlschema }}\" itemprop=\"datePublished\">{{ page.date | date: \"%b %-d, %Y\" }}</time>{% if page.author %} • <span itemprop=\"author\" itemscope itemtype=\"http://schema.org/Person\"><span itemprop=\"name\">{{ page.author }}</span></span>{% endif %}</p>
  </header>

  <div class=\"post-content\" itemprop=\"articleBody\">
    {{ content }}
  </div>

</article>
<article class=\"post\" itemscope itemtype=\"http://schema.org/BlogPosting\">

  <header class=\"post-header\">
    <h1 class=\"post-title\" itemprop=\"name headline\">Welcome to Jekyll!</h1>
    <p class=\"post-meta\"><time datetime=\"2016-08-17T23:50:36-04:00\" itemprop=\"datePublished\">Aug 17, 2016</time></p>
  </header>

  <div class=\"post-content\" itemprop=\"articleBody\">
    <p>You’ll find this post in your <code class=\"highlighter-rouge\">_posts</code> directory. Go ahead and edit it and re-build the site to see your changes. You can rebuild the site in many different ways, but the most common way is to run <code class=\"highlighter-rouge\">jekyll serve</code>, which launches a web server and auto-regenerates your site when a file is updated.</p>

<p>To add new posts, simply add a file in the <code class=\"highlighter-rouge\">_posts</code> directory that follows the convention <code class=\"highlighter-rouge\">YYYY-MM-DD-name-of-post.ext</code> and includes the necessary front matter. Take a look at the source for this post to get an idea about how it works.</p>

<p>Jekyll also offers powerful support for code snippets:</p>

<figure class=\"highlight\"><pre><code class=\"language-ruby\" data-lang=\"ruby\"><span class=\"k\">def</span> <span class=\"nf\">print_hi</span><span class=\"p\">(</span><span class=\"nb\">name</span><span class=\"p\">)</span>
  <span class=\"nb\">puts</span> <span class=\"s2\">\"Hi, </span><span class=\"si\">\#{</span><span class=\"nb\">name</span><span class=\"si\">}</span><span class=\"s2\">\"</span>
<span class=\"k\">end</span>
<span class=\"n\">print_hi</span><span class=\"p\">(</span><span class=\"s1\">'Tom'</span><span class=\"p\">)</span>
<span class=\"c1\">#=&gt; prints 'Hi, Tom' to STDOUT.</span></code></pre></figure>

<p>Check out the <a href=\"http://jekyllrb.com/docs/home\">Jekyll docs</a> for more info on how to get the most out of Jekyll. File all bugs/feature requests at <a href=\"https://github.com/jekyll/jekyll\">Jekyll’s GitHub repo</a>. If you have questions, you can ask them on <a href=\"https://talk.jekyllrb.com/\">Jekyll Talk</a>.</p>


  </div>


</article>

This is just one example of a converter and a layout, kramdown for markdown and the above layout respectively. Jekyll has other built in converters like smartypants. Jekyll also includes jekyll-sass-converter by default in its gemspec, which is not a built in converter, but you’ll find when you run jekyll new. You can install other converters from plugins. Layouts can come from files in your _layouts folder or Jekyll templates you install as gems.

Jekyll’s core

Now that you understand Jekyll’s transformation, let’s look at how it fits into the larger process of building a site from input files.

If we follow the code through an invocation of jekyll build, we find site.process at the core of Jekyll. You can find the important parts below with my added explanatory comments. See debugging if you want to walk through the invocation yourself.

# Public: Read, process, and write this Site to output.
#
# Returns nothing.
def process
  reset # empty all the site's layouts/pages/static_file/data/collection lists.
  read # read your directory's files into the site object, classifying them depending on 
  # where they're found, what they contain, and what you've configured.
  generate # Giving you a chance to run generators to add more stuff to the site.
  render # this is what runs our transformation.
  cleanup # get rid of any files that might be from an old build or are processing metadata.
  write # write files out to your destination folder.
  print_stats
end

render applies Jekyll’s transformation to files with yaml frontmatter. Docs are created from collection files with yaml frontmatter, and pages are any other files with yaml frontmatter. You have to specify collections in your _config.yml, so you’ll know if you have them. You also need to know that posts and drafts are just special kinds of collections, so these are also docs.

def render
    ...
    render_docs(payload) # this renders collection files that have frontmatter, including _posts (drafts are merged with posts with -D)
    render_pages(payload) # this renders non collection files that have frontmatter
    # these might be your `feed.xml`, `index.html`, `main.scss`, etc.
    ...
end
...

def render_docs(payload)
  collections.each do |_, collection|
    collection.docs.each do |document|
      if regenerator.regenerate?(document)
        document.output = Jekyll::Renderer.new(self, document, payload).run
        document.trigger_hooks(:post_render)
      end
    end
  end
end
...
def render_pages(payload)
  pages.flatten.each do |page|
    if regenerator.regenerate?(page)
      page.output = Jekyll::Renderer.new(self, page, payload).run
      page.trigger_hooks(:post_render)
    end
  end
end

Testing a complete build

Let’s look at a sample outcome of building with some types of files in different directories, with and without -D. You can find this setup as a test container at bytesandwich/jekyll-outcomes.

It puts a copy these files:

  • 2016-05-05-post-normal.md # a normal post, dated in the past
  • 2016-05-05-post-without-frontmatter.md # a post with no frontmatter, dated in the past
  • 2020-02-02-post-future.md # a normal post, dated in the future
  • frontmatter-not-post.md # a frontmatter file that isn’t a post
  • text.txt # a normal text file
  • yaml.yml # a normal yaml file

… into each of these dirs:

  • /
  • /_posts
  • /_drafts
  • /_data
  • /_my_output_collection
  • /_my_non_output_collection
  • /_underscore_dir
  • /regular_dir

Except, for each file x dir pair, it appends the dir’s last node’s name onto the end of the filename, so that we can track back which output file came from which input directory. You can see this in the tree output below the tables.

I’ve made a table of origin row:directory and col:file, and the cell contains the file’s outcomes which are:

  • copied without changes
  • omitted
  • transformed and put into the matching directory
  • post transformed which is transformed and then put into a special directory layout of nested dates.

File output without drafts

  text.txt frontmatter-not-post.md 2016-05-05-post-without-frontmatter.md yaml.yml 2020-02-02-post-future.md 2016-05-05-post-normal.md
/ copied transformed copied copied transformed transformed
/posts omitted omitted post transformed omitted omitted post transformed
/drafts omitted omitted omitted omitted omitted omitted
/data omitted omitted omitted omitted omitted omitted
/my_output_collection copied transformed copied copied omitted transformed
/my_non_output_collection copied omitted copied copied omitted omitted
/underscore_dir omitted omitted omitted omitted omitted omitted
/regular_dir copied transformed copied copied transformed transformed


File output with drafts

  text.txt frontmatter-not-post.md 2016-05-05-post-without-frontmatter.md yaml.yml 2020-02-02-post-future.md 2016-05-05-post-normal.md
/ copied transformed copied copied transformed transformed
/posts omitted omitted post transformed omitted omitted post transformed
/drafts None post transformed post transformed post transformed omitted post transformed
/data omitted omitted omitted omitted omitted omitted
/my_output_collection copied transformed copied copied omitted transformed
/my_non_output_collection copied omitted copied copied omitted omitted
/underscore_dir omitted omitted omitted omitted omitted omitted
/regular_dir copied transformed copied copied transformed transformed
source
site
site with drafts
bash-4.3# tree .
.
├── 2016-05-05-post-normal.md
├── 2016-05-05-post-without-frontmatter.md
├── 2020-02-02-post-future.md
├── Gemfile
├── Gemfile.lock
├── _config.yml
├── _data
│   ├── 2016-05-05-post-normal-data.md
│   ├── 2016-05-05-post-without-frontmatter-data.md
│   ├── 2020-02-02-post-future-data.md
│   ├── frontmatter-not-post-data.md
│   ├── text-data.txt
│   └── yaml-data.yml
├── _drafts
│   ├── 2016-05-05-post-normal-drafts.md
│   ├── 2016-05-05-post-without-frontmatter-drafts.md
│   ├── 2020-02-02-post-future-drafts.md
│   ├── frontmatter-not-post-drafts.md
│   ├── text-drafts.txt
│   └── yaml-drafts.yml
├── _layouts
│   └── special.html
├── _my_non_output_collection
│   ├── 2016-05-05-post-normal-my_non_output_collection.md
│   ├── 2016-05-05-post-without-frontmatter-my_non_output_collection.md
│   ├── 2020-02-02-post-future-my_non_output_collection.md
│   ├── frontmatter-not-post-my_non_output_collection.md
│   ├── text-my_non_output_collection.txt
│   └── yaml-my_non_output_collection.yml
├── _my_output_collection
│   ├── 2016-05-05-post-normal-my_output_collection.md
│   ├── 2016-05-05-post-without-frontmatter-my_output_collection.md
│   ├── 2020-02-02-post-future-my_output_collection.md
│   ├── frontmatter-not-post-my_output_collection.md
│   ├── text-my_output_collection.txt
│   └── yaml-my_output_collection.yml
├── _posts
│   ├── 2016-05-05-post-normal-posts.md
│   ├── 2016-05-05-post-without-frontmatter-posts.md
│   ├── 2016-09-14-welcome-to-jekyll.markdown
│   ├── 2020-02-02-post-future-posts.md
│   ├── frontmatter-not-post-posts.md
│   ├── text-posts.txt
│   └── yaml-posts.yml
├── _underscore_dir
│   ├── 2016-05-05-post-normal-underscore_dir.md
│   ├── 2016-05-05-post-without-frontmatter-underscore_dir.md
│   ├── 2020-02-02-post-future-underscore_dir.md
│   ├── frontmatter-not-post-underscore_dir.md
│   ├── text-underscore_dir.txt
│   └── yaml-underscore_dir.yml
├── about.md
├── css
│   └── main.scss
├── feed.xml
├── frontmatter-not-post.md
├── index.html
├── regular_dir
│   ├── 2016-05-05-post-normal-regular_dir.md
│   ├── 2016-05-05-post-without-frontmatter-regular_dir.md
│   ├── 2020-02-02-post-future-regular_dir.md
│   ├── frontmatter-not-post-regular_dir.md
│   ├── text-regular_dir.txt
│   └── yaml-regular_dir.yml
├── text.txt
└── yaml.yml

9 directories, 57 files
bash-4.3# tree _site
_site
├── 2016
│   └── 05
│       └── 05
│           ├── post-normal-posts.html
│           └── post-without-frontmatter-posts.html
├── 2016-05-05-post-normal.html
├── 2016-05-05-post-without-frontmatter.md
├── 2020-02-02-post-future.html
├── Gemfile
├── Gemfile.lock
├── about
│   └── index.html
├── css
│   └── main.css
├── feed.xml
├── frontmatter-not-post.html
├── index.html
├── jekyll
│   └── update
│       └── 2016
│           └── 09
│               └── 14
│                   └── welcome-to-jekyll.html
├── my_non_output_collection
│   ├── 2016-05-05-post-without-frontmatter-my_non_output_collection.md
│   ├── text-my_non_output_collection.txt
│   └── yaml-my_non_output_collection.yml
├── my_output_collection
│   ├── 2016-05-05-post-normal-my_output_collection.html
│   ├── 2016-05-05-post-without-frontmatter-my_output_collection.md
│   ├── frontmatter-not-post-my_output_collection.html
│   ├── text-my_output_collection.txt
│   └── yaml-my_output_collection.yml
├── regular_dir
│   ├── 2016-05-05-post-normal-regular_dir.html
│   ├── 2016-05-05-post-without-frontmatter-regular_dir.md
│   ├── 2020-02-02-post-future-regular_dir.html
│   ├── frontmatter-not-post-regular_dir.html
│   ├── text-regular_dir.txt
│   └── yaml-regular_dir.yml
├── text.txt
└── yaml.yml

13 directories, 29 files
bash-4.3# tree _site
_site
├── 2016
│   ├── 05
│   │   └── 05
│   │       ├── post-normal-drafts.html
│   │       ├── post-normal-posts.html
│   │       ├── post-without-frontmatter-drafts.html
│   │       └── post-without-frontmatter-posts.html
│   └── 09
│       └── 14
│           ├── frontmatter-not-post-drafts.html
│           ├── text-drafts.txt
│           └── yaml-drafts.yml
├── 2016-05-05-post-normal.html
├── 2016-05-05-post-without-frontmatter.md
├── 2020-02-02-post-future.html
├── Gemfile
├── Gemfile.lock
├── about
│   └── index.html
├── css
│   └── main.css
├── feed.xml
├── frontmatter-not-post.html
├── index.html
├── jekyll
│   └── update
│       └── 2016
│           └── 09
│               └── 14
│                   └── welcome-to-jekyll.html
├── my_non_output_collection
│   ├── 2016-05-05-post-without-frontmatter-my_non_output_collection.md
│   ├── text-my_non_output_collection.txt
│   └── yaml-my_non_output_collection.yml
├── my_output_collection
│   ├── 2016-05-05-post-normal-my_output_collection.html
│   ├── 2016-05-05-post-without-frontmatter-my_output_collection.md
│   ├── frontmatter-not-post-my_output_collection.html
│   ├── text-my_output_collection.txt
│   └── yaml-my_output_collection.yml
├── regular_dir
│   ├── 2016-05-05-post-normal-regular_dir.html
│   ├── 2016-05-05-post-without-frontmatter-regular_dir.md
│   ├── 2020-02-02-post-future-regular_dir.html
│   ├── frontmatter-not-post-regular_dir.html
│   ├── text-regular_dir.txt
│   └── yaml-regular_dir.yml
├── text.txt
└── yaml.yml

15 directories, 34 files

Jekyll’s objects: Posts, Drafts, Pages, Data, Collections, Layouts, & Includes

The best place to continue to learn is jekyll directory structure, where you can find further descriptions of the types of files in these directories.

Debugging

(at least on osx) In terminal the jekyll executable is a RubyGem generator wrapper script that calls /Library/Ruby/Gems/2.0.0/gems/jekyll-3.2.1/exe/jekyll where we can start debugging with pry-byebug if we add two lines, that you’ll find I’ve added at 8 and 10 below:

From: /Library/Ruby/Gems/2.0.0/gems/jekyll-3.2.1/exe/jekyll @ line 12 :

     7: require "mercenary"
     8: require "pry-byebug"
     9:
    10: binding.pry
    11:
 => 12: Jekyll::PluginManager.require_from_bundler
    13:
    14: Jekyll::Deprecator.process(ARGV)
    15:
    16: Mercenary.program(:jekyll) do |p|
    17:   p.version Jekyll::VERSION

[1] pry(main)>break /Library/Ruby/Gems/2.0.0/gems/jekyll-3.2.1/lib/jekyll/site.rb:66

Jekyll has its own command line interface framework, Mercenary, which will

Mercenary.program(:jekyll) do |p|
  ...
  p.command(:build) do |c|
    ...
    Jekyll::Commands::Build.process(options)
    ...
  end

invokes build(site, options) in build.process which invokes process_site

which sets up the default configuration, establishing posts as a collection as we learned above.

process_site invokes site.process.

  1. When Jekyll omits a file, it may read the file in as data you can access through liquid template arguments in other file’s liquid templating.

  2. This may very well change in this issue to omit frontmatter.