Commit 3f5d7c7e authored by Guillaume Grossetie's avatar Guillaume Grossetie Committed by James Lopez

Add basic support for AsciiDoc include directive

See http://asciidoctor.org/docs/user-manual/#include-directive
parent cd300323
......@@ -130,6 +130,7 @@ gem 'org-ruby', '~> 0.9.12'
gem 'creole', '~> 0.5.0'
gem 'wikicloth', '0.8.1'
gem 'asciidoctor', '~> 1.5.8'
gem 'asciidoctor-include-ext', '~> 0.3.1', require: false
gem 'asciidoctor-plantuml', '0.0.8'
gem 'rouge', '~> 3.1'
gem 'truncato', '~> 0.7.11'
......
......@@ -67,6 +67,8 @@ GEM
faraday_middleware-multi_json (~> 0.0)
oauth2 (~> 1.0)
asciidoctor (1.5.8)
asciidoctor-include-ext (0.3.1)
asciidoctor (>= 1.5.6, < 3.0.0)
asciidoctor-plantuml (0.0.8)
asciidoctor (~> 1.5)
ast (2.4.0)
......@@ -1024,6 +1026,7 @@ DEPENDENCIES
apollo_upload_server (~> 2.0.0.beta3)
asana (~> 0.8.1)
asciidoctor (~> 1.5.8)
asciidoctor-include-ext (~> 0.3.1)
asciidoctor-plantuml (= 0.0.8)
attr_encrypted (~> 3.1.0)
awesome_print
......
......@@ -263,6 +263,11 @@ module MarkupHelper
end
def asciidoc_unsafe(text, context = {})
context.merge!(
commit: @commit,
ref: @ref,
requested_path: @path
)
Gitlab::Asciidoc.render(text, context)
end
......
---
title: Add support for AsciiDoc include directive
merge_request: 28417
author: "Jakub Jirutka & Guillaume Grossetie"
type: added
# AsciiDoc
GitLab uses the [Asciidoctor](https://asciidoctor.org) gem to convert AsciiDoc content to HTML5.
Consult the [Asciidoctor User Manual](https://asciidoctor.org/docs/user-manual) for a complete Asciidoctor reference.
## Syntax
Here's a brief reference of the most commonly used AsciiDoc syntax.
You can find the full documentation for the AsciiDoc syntax at https://asciidoctor.org/docs.
### Paragraphs
```asciidoc
A normal paragraph.
Line breaks are not preserved.
```
Line comments, which are lines that start with `//`, are skipped:
```
// this is a comment
```
A blank line separates paragraphs.
A paragraph with the `[%hardbreaks]` option will preserve line breaks:
```asciidoc
[%hardbreaks]
This paragraph carries the `hardbreaks` option.
Notice how line breaks are now preserved.
```
An indented (literal) paragraph disables text formatting,
preserves spaces and line breaks, and is displayed in a
monospaced font:
```asciidoc
This literal paragraph is indented with one space.
As a consequence, *text formatting*, spaces,
and lines breaks will be preserved.
```
An admonition paragraph grabs the reader's attention:
```asciidoc
NOTE: This is a brief reference, please read the full documentation at https://asciidoctor.org/docs.
TIP: Lists can be indented. Leading whitespace is not significant.
```
### Text Formatting
**Constrained (applied at word boundaries)**
```asciidoc
*strong importance* (aka bold)
_stress emphasis_ (aka italic)
`monospaced` (aka typewriter text)
"`double`" and '`single`' typographic quotes
+passthrough text+ (substitutions disabled)
`+literal text+` (monospaced with substitutions disabled)
```
**Unconstrained (applied anywhere)**
```asciidoc
**C**reate+**R**ead+**U**pdate+**D**elete
fan__freakin__tastic
``mono``culture
```
**Replacements**
```asciidoc
A long time ago in a galaxy far, far away...
(C) 1976 Arty Artisan
I believe I shall--no, actually I won't.
```
**Macros**
```asciidoc
// where c=specialchars, q=quotes, a=attributes, r=replacements, m=macros, p=post_replacements, etc.
The European icon:flag[role=blue] is blue & contains pass:[************] arranged in a icon:circle-o[role=yellow].
The pass:c[->] operator is often referred to as the stabby lambda.
Since `pass:[++]` has strong priority in AsciiDoc, you can rewrite pass:c,a,r[C++ => C{pp}].
// activate stem support by adding `:stem:` to the document header
stem:[sqrt(4) = 2]
```
### Attributes
```asciidoc
// define attributes in the document header
:name: value
```
```asciidoc
:url-gem: https://rubygems.org/gems/asciidoctor
You can download and install Asciidoctor {asciidoctor-version} from {url-gem}.
C{pp} is not required, only Ruby.
Use a leading backslash to output a word enclosed in curly braces, like \{name}.
```
### Links
```asciidoc
https://example.org/page[A webpage]
link:../path/to/file.txt[A local file]
xref:document.adoc[A sibling document]
mailto:hello@example.org[Email to say hello!]
```
### Anchors
```asciidoc
[[idname,reference text]]
// or written using normal block attributes as `[#idname,reftext=reference text]`
A paragraph (or any block) with an anchor (aka ID) and reftext.
See <<idname>> or <<idname,optional text of internal link>>.
xref:document.adoc#idname[Jumps to anchor in another document].
This paragraph has a footnote.footnote:[This is the text of the footnote.]
```
### Lists
#### Unordered
```asciidoc
* level 1
** level 2
*** level 3
**** level 4
***** etc.
* back at level 1
+
Attach a block or paragraph to a list item using a list continuation (which you can enclose in an open block).
.Some Authors
[circle]
- Edgar Allen Poe
- Sheri S. Tepper
- Bill Bryson
```
#### Ordered
```asciidoc
. Step 1
. Step 2
.. Step 2a
.. Step 2b
. Step 3
.Remember your Roman numerals?
[upperroman]
. is one
. is two
. is three
```
#### Checklist
```asciidoc
* [x] checked
* [ ] not checked
```
#### Callout
```asciidoc
// enable callout bubbles by adding `:icons: font` to the document header
[,ruby]
----
puts 'Hello, World!' # <1>
----
<1> Prints `Hello, World!` to the console.
```
#### Description
```asciidoc
first term:: description of first term
second term::
description of second term
```
### Document Structure
#### Header
```asciidoc
= Document Title
Author Name <author@example.org>
v1.0, 2019-01-01
```
#### Sections
```asciidoc
= Document Title (Level 0)
== Level 1
=== Level 2
==== Level 3
===== Level 4
====== Level 5
== Back at Level 1
```
#### Includes
```asciidoc
include::basics.adoc[]
// define -a allow-uri-read to allow content to be read from URI
include::https://example.org/installation.adoc[]
```
### Blocks
```asciidoc
--
open - a general-purpose content wrapper; useful for enclosing content to attach to a list item
--
```
```asciidoc
// recognized types include CAUTION, IMPORTANT, NOTE, TIP, and WARNING
// enable admonition icons by setting `:icons: font` in the document header
[NOTE]
====
admonition - a notice for the reader, ranging in severity from a tip to an alert
====
```
```asciidoc
====
example - a demonstration of the concept being documented
====
```
```asciidoc
.Toggle Me
[%collapsible]
====
collapsible - these details are revealed by clicking the title
====
```
```asciidoc
****
sidebar - auxiliary content that can be read independently of the main content
****
```
```asciidoc
....
literal - an exhibit that features program output
....
```
```asciidoc
----
listing - an exhibit that features program input, source code, or the contents of a file
----
```
```asciidoc
[,language]
----
source - a listing that is embellished with (colorized) syntax highlighting
----
```
```asciidoc
\```language
fenced code - a shorthand syntax for the source block
\```
```
```asciidoc
[,attribution,citetitle]
____
quote - a quotation or excerpt; attribution with title of source are optional
____
```
```asciidoc
[verse,attribution,citetitle]
____
verse - a literary excerpt, often a poem; attribution with title of source are optional
____
```
```asciidoc
++++
pass - content passed directly to the output document; often raw HTML
++++
```
```asciidoc
// activate stem support by adding `:stem:` to the document header
[stem]
++++
x = y^2
++++
```
```asciidoc
////
comment - content which is not included in the output document
////
```
### Tables
```asciidoc
.Table Attributes
[cols=>1h;2d,width=50%,frame=topbot]
|===
| Attribute Name | Values
| options
| header,footer,autowidth
| cols
| colspec[;colspec;...]
| grid
| all \| cols \| rows \| none
| frame
| all \| sides \| topbot \| none
| stripes
| all \| even \| odd \| none
| width
| (0%..100%)
| format
| psv {vbar} csv {vbar} dsv
|===
```
### Multimedia
```asciidoc
image::screenshot.png[block image,800,450]
Press image:reload.svg[reload,16,opts=interactive] to reload the page.
video::movie.mp4[width=640,start=60,end=140,options=autoplay]
video::aHjpOzsQ9YI[youtube]
video::300817511[vimeo]
```
### Breaks
```asciidoc
// thematic break (aka horizontal rule)
---
```
```asciidoc
// page break
<<<
```
......@@ -68,7 +68,7 @@ according to the markup language.
| Plain text | `txt` |
| [Markdown](../../markdown.md) | `mdown`, `mkd`, `mkdn`, `md`, `markdown` |
| [reStructuredText](http://docutils.sourceforge.net/rst.html) | `rst` |
| [Asciidoc](https://asciidoctor.org/docs/what-is-asciidoc/) | `adoc`, `ad`, `asciidoc` |
| [AsciiDoc](../../asciidoc.md) | `adoc`, `ad`, `asciidoc` |
| [Textile](https://txstyle.org/) | `textile` |
| [rdoc](http://rdoc.sourceforge.net/doc/index.html) | `rdoc` |
| [Orgmode](https://orgmode.org/) | `org` |
......
# frozen_string_literal: true
require 'asciidoctor'
require 'asciidoctor/converter/html5'
require "asciidoctor-plantuml"
require 'asciidoctor-plantuml'
require 'asciidoctor/extensions'
require 'gitlab/asciidoc/html5_converter'
module Gitlab
# Parser/renderer for the AsciiDoc format that uses Asciidoctor and filters
# the resulting HTML through HTML pipeline filters.
module Asciidoc
DEFAULT_ADOC_ATTRS = [
'showtitle', 'idprefix=user-content-', 'idseparator=-', 'env=gitlab',
'env-gitlab', 'source-highlighter=html-pipeline', 'icons=font',
'outfilesuffix=.adoc'
].freeze
MAX_INCLUDE_DEPTH = 5
DEFAULT_ADOC_ATTRS = {
'showtitle' => true,
'idprefix' => 'user-content-',
'idseparator' => '-',
'env' => 'gitlab',
'env-gitlab' => '',
'source-highlighter' => 'html-pipeline',
'icons' => 'font',
'outfilesuffix' => '.adoc',
'max-include-depth' => MAX_INCLUDE_DEPTH
}.freeze
# Public: Converts the provided Asciidoc markup into HTML.
#
# input - the source text in Asciidoc format
# context - :commit, :project, :ref, :requested_path
#
def self.render(input, context)
extensions = proc do
include_processor ::Gitlab::Asciidoc::IncludeProcessor.new(context)
end
asciidoc_opts = { safe: :secure,
backend: :gitlab_html5,
attributes: DEFAULT_ADOC_ATTRS }
attributes: DEFAULT_ADOC_ATTRS,
extensions: extensions }
context[:pipeline] = :ascii_doc
......@@ -40,29 +54,5 @@ module Gitlab
conf.txt_enable = false
end
end
class Html5Converter < Asciidoctor::Converter::Html5Converter
extend Asciidoctor::Converter::Config
register_for 'gitlab_html5'
def stem(node)
return super unless node.style.to_sym == :latexmath
%(<pre#{id_attribute(node)} data-math-style="display"><code>#{node.content}</code></pre>)
end
def inline_quoted(node)
return super unless node.type.to_sym == :latexmath
%(<code#{id_attribute(node)} data-math-style="inline">#{node.text}</code>)
end
private
def id_attribute(node)
node.id ? %( id="#{node.id}") : nil
end
end
end
end
# frozen_string_literal: true
require 'asciidoctor'
require 'asciidoctor/converter/html5'
module Gitlab
module Asciidoc
class Html5Converter < Asciidoctor::Converter::Html5Converter
extend Asciidoctor::Converter::Config
register_for 'gitlab_html5'
def stem(node)
return super unless node.style.to_sym == :latexmath
%(<pre#{id_attribute(node)} data-math-style="display"><code>#{node.content}</code></pre>)
end
def inline_quoted(node)
return super unless node.type.to_sym == :latexmath
%(<code#{id_attribute(node)} data-math-style="inline">#{node.text}</code>)
end
private
def id_attribute(node)
node.id ? %( id="#{node.id}") : nil
end
end
end
end
# frozen_string_literal: true
require 'asciidoctor/include_ext/include_processor'
module Gitlab
module Asciidoc
# Asciidoctor extension for processing includes (macro include::[]) within
# documents inside the same repository.
class IncludeProcessor < Asciidoctor::IncludeExt::IncludeProcessor
extend ::Gitlab::Utils::Override
def initialize(context)
super(logger: Gitlab::AppLogger)
@context = context
@repository = context[:project].try(:repository)
# Note: Asciidoctor calls #freeze on extensions, so we can't set new
# instance variables after initialization.
@cache = {
uri_types: {}
}
end
protected
override :include_allowed?
def include_allowed?(target, reader)
doc = reader.document
return false if doc.attributes.fetch('max-include-depth').to_i < 1
return false if target_uri?(target)
true
end
override :resolve_target_path
def resolve_target_path(target, reader)
return unless repository.try(:exists?)
base_path = reader.include_stack.empty? ? requested_path : reader.file
path = resolve_relative_path(target, base_path)
path if Gitlab::Git::Blob.find(repository, ref, path)
end
override :read_lines
def read_lines(filename, selector)
blob = read_blob(ref, filename)
if selector
blob.data.each_line.select.with_index(1, &selector)
else
blob.data
end
end
override :unresolved_include!
def unresolved_include!(target, reader)
reader.unshift_line("*[ERROR: include::#{target}[] - unresolved directive]*")
end
private
attr_accessor :context, :repository, :cache
# Gets a Blob at a path for a specific revision.
# This method will check that the Blob exists and contains readable text.
#
# revision - The String SHA1.
# path - The String file path.
#
# Returns a Blob
def read_blob(ref, filename)
blob = repository&.blob_at(ref, filename)
raise 'Blob not found' unless blob
raise 'File is not readable' unless blob.readable_text?
blob
end
# Resolves the given relative path of file in repository into canonical
# path based on the specified base_path.
#
# Examples:
#
# # File in the same directory as the current path
# resolve_relative_path("users.adoc", "doc/api/README.adoc")
# # => "doc/api/users.adoc"
#
# # File in the same directory, which is also the current path
# resolve_relative_path("users.adoc", "doc/api")
# # => "doc/api/users.adoc"
#
# # Going up one level to a different directory
# resolve_relative_path("../update/7.14-to-8.0.adoc", "doc/api/README.adoc")
# # => "doc/update/7.14-to-8.0.adoc"
#
# Returns a String
def resolve_relative_path(path, base_path)
p = Pathname(base_path)
p = p.dirname unless p.extname.empty?
p += path
p.cleanpath.to_s
end
def current_commit
cache[:current_commit] ||= context[:commit] || repository&.commit(ref)
end
def ref
context[:ref] || context[:project].default_branch
end
def requested_path
cache[:requested_path] ||= Addressable::URI.unescape(context[:requested_path])
end
def uri_type(path)
cache[:uri_types][path] ||= current_commit&.uri_type(path)
end
end
end
end
......@@ -3,20 +3,23 @@ require 'nokogiri'
module Gitlab
describe Asciidoc do
let(:input) { '<b>ascii</b>' }
let(:context) { {} }
let(:html) { 'H<sub>2</sub>O' }
include FakeBlobHelpers
before do
allow_any_instance_of(ApplicationSetting).to receive(:current).and_return(::ApplicationSetting.create_from_defaults)
end
context "without project" do
before do
allow_any_instance_of(ApplicationSetting).to receive(:current).and_return(::ApplicationSetting.create_from_defaults)
end
let(:input) { '<b>ascii</b>' }
let(:context) { {} }
let(:html) { 'H<sub>2</sub>O' }
it "converts the input using Asciidoctor and default options" do
expected_asciidoc_opts = {
safe: :secure,
backend: :gitlab_html5,
attributes: described_class::DEFAULT_ADOC_ATTRS
attributes: described_class::DEFAULT_ADOC_ATTRS,
extensions: be_a(Proc)
}
expect(Asciidoctor).to receive(:convert)
......@@ -30,7 +33,8 @@ module Gitlab
expected_asciidoc_opts = {
safe: :secure,
backend: :gitlab_html5,
attributes: described_class::DEFAULT_ADOC_ATTRS
attributes: described_class::DEFAULT_ADOC_ATTRS,
extensions: be_a(Proc)
}
expect(Asciidoctor).to receive(:convert)
......@@ -105,6 +109,174 @@ module Gitlab
end
end
context 'with project' do
let(:context) do
{
commit: commit,
project: project,
ref: ref,
requested_path: requested_path
}
end
let(:commit) { project.commit(ref) }
let(:project) { create(:project, :repository) }
let(:ref) { 'asciidoc' }
let(:requested_path) { '/' }
context 'include directive' do
subject(:output) { render(input, context) }
let(:input) { "Include this:\n\ninclude::#{include_path}[]" }
before do
current_file = requested_path
current_file += 'README.adoc' if requested_path.end_with? '/'
create_file(current_file, "= AsciiDoc\n")
end
context 'with path to non-existing file' do
let(:include_path) { 'not-exists.adoc' }
it 'renders Unresolved directive placeholder' do
is_expected.to include("<strong>[ERROR: include::#{include_path}[] - unresolved directive]</strong>")
end
end
shared_examples :invalid_include do
let(:include_path) { 'dk.png' }
before do
allow(project.repository).to receive(:blob_at).and_return(blob)
end
it 'does not read the blob' do
expect(blob).not_to receive(:data)
end
it 'renders Unresolved directive placeholder' do
is_expected.to include("<strong>[ERROR: include::#{include_path}[] - unresolved directive]</strong>")
end
end
context 'with path to a binary file' do
let(:blob) { fake_blob(path: 'dk.png', binary: true) }
include_examples :invalid_include
end
context 'with path to file in external storage' do
let(:blob) { fake_blob(path: 'dk.png', lfs: true) }
before do
allow(Gitlab.config.lfs).to receive(:enabled).and_return(true)
project.update_attribute(:lfs_enabled, true)
end
include_examples :invalid_include
end
context 'with path to a textual file' do
let(:include_path) { 'sample.adoc' }
before do
create_file(file_path, "Content from #{include_path}")
end
shared_examples :valid_include do
[
['/doc/sample.adoc', 'doc/sample.adoc', 'absolute path'],
['sample.adoc', 'doc/api/sample.adoc', 'relative path'],
['./sample.adoc', 'doc/api/sample.adoc', 'relative path with leading ./'],
['../sample.adoc', 'doc/sample.adoc', 'relative path to a file up one directory'],
['../../sample.adoc', 'sample.adoc', 'relative path for a file up multiple directories']
].each do |include_path_, file_path_, desc|
context "the file is specified by #{desc}" do
let(:include_path) { include_path_ }
let(:file_path) { file_path_ }
it 'includes content of the file' do
is_expected.to include('<p>Include this:</p>')
is_expected.to include("<p>Content from #{include_path}</p>")
end
end
end
end
context 'when requested path is a file in the repo' do
let(:requested_path) { 'doc/api/README.adoc' }
include_examples :valid_include
context 'without a commit (only ref)' do
let(:commit) { nil }
include_examples :valid_include
end
end
context 'when requested path is a directory in the repo' do
let(:requested_path) { 'doc/api/' }
include_examples :valid_include
context 'without a commit (only ref)' do
let(:commit) { nil }
include_examples :valid_include
end
end
end
context 'recursive includes with relative paths' do
let(:input) do
<<~ADOC
Source: requested file
include::doc/README.adoc[]
include::license.adoc[]
ADOC
end
before do
create_file 'doc/README.adoc', <<~ADOC
Source: doc/README.adoc
include::../license.adoc[]
include::api/hello.adoc[]
ADOC
create_file 'license.adoc', <<~ADOC
Source: license.adoc
ADOC
create_file 'doc/api/hello.adoc', <<~ADOC
Source: doc/api/hello.adoc
include::./common.adoc[]
ADOC
create_file 'doc/api/common.adoc', <<~ADOC
Source: doc/api/common.adoc
ADOC
end
it 'includes content of the included files recursively' do
expect(output.gsub(/<[^>]+>/, '').gsub(/\n\s*/, "\n").strip).to eq <<~ADOC.strip
Source: requested file
Source: doc/README.adoc
Source: license.adoc
Source: doc/api/hello.adoc
Source: doc/api/common.adoc
Source: license.adoc
ADOC
end
end
def create_file(path, content)
project.repository.create_file(project.creator, path, content,
message: "Add #{path}", branch_name: 'asciidoc')
end
end
end
def render(*args)
described_class.render(*args)
end
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment