Commit Graph

41 Commits

Author SHA1 Message Date
Anders Kaseorg d9f12cfb49 documentation_crawler: They’re called fragments, not permalinks.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg 4f28b88746 documentation_crawler: Set a 15 second download timeout.
By default it just hangs for as long as the server keeps the
connection open.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg c59747360a documentation_crawler: Set the default log level to warning.
This doesn’t seem to add any noise in the normal case, but if anything
shows up here we might want to see it.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg 2f547eaee4 documentation_crawler: Improve error handling.
* Remove the custom has_error logic in favor of checking whether any
  errors were logged, which gives us a much better chance at catching
  unanticipated exceptions.
* Use our error_callback for the initial requests of start_urls too.
* Clean up mypy types.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
sumanthvrao c5de6d0e18 spiders: Set error status for exceptions generated by spider.
Spider raises exceptions when errors like FileNotFound
are detected. However, these did not set error state
before exiting causing spider to fail silently.

This patch sets the status causing exceptions to exit with
non-zero exit status.
2019-04-08 17:04:29 -07:00
Anders Kaseorg ee8ff4df66 tools: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:10:31 -08:00
Tim Abbott 5a8ad84381 test-documentation: Exclude the 500ing Citizen Code of Conduct.
We might want to eventually remove this, but I'm holding out hope that
it'll be fixed.
2018-12-29 15:30:19 -08:00
Vishnu Ks 6f5b1903c4 docs: Update code-reviewing docs to reflect CI changes. 2018-12-29 15:28:26 -08:00
Sumanth V Rao 776b981dde test-help-documentation: Skips checking external links.
This option causes test-help-documentation to verify internal links only.
This prevents documentation from flaking in CI due to links from
external websites having failed in the documentation.

Based on conversation:
https://chat.zulip.org/#narrow/stream/43-automated-testing/subject/quora.20404/near/674997
2018-12-29 20:15:30 +05:30
Sumanth V Rao e1f33e0f6b test-documentation: Add --skip-external-links option.
This option causes test-documentation to only verify internal links
that we control and are important to be correct.  This prevents
test-documentation from flaking in CI due to issues with the dozens of
third-party blocks that we link to from various parts of our
documentation.

Tweaked by tabbott for comment clarity, and to also include
github.com/zulip links.

Fixes #10942.
2018-12-05 12:39:59 -08:00
Tim Abbott 723fb60476 docs: Fix up some broken links output spam. 2018-11-28 09:49:39 -08:00
Rishi Gupta 69e9a8f196 user docs: Add a few portico URLs to documentation crawler test. 2018-11-01 11:59:50 -07:00
Tim Abbott d9887759b4 plans: Fix broken link from FAQ to import docs and add tests. 2018-10-31 11:44:21 -07:00
Anders Kaseorg 037f696d26 Enable pycodestyle W605 (invalid escape sequence).
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are

zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'

which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Eeshan Garg ca5ea20ab7 doc tests: Add portico pages to tools/test-help-documentation.
Fixes #9117.
2018-04-21 21:51:32 -07:00
Tim Abbott b0b134cb4c help: Clean up settings_html and subscriptions_html.
After some thinking, I don't think there's any actual value to doing
the ../ style relative links here, whereas there is actual harm from
the links being slightly broken in the current model.  We fix this by
just using /#settings as the URL.

Fixes #8978.
2018-04-05 14:48:26 -07:00
Tim Abbott ed5b374ffa help docs: Fix a sorta buggy URL test. 2018-04-05 13:07:34 -07:00
David Rosa Tamsen 7072fa5b37 docs: Reorganize developer docs to improve navigation.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.

This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.

Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).

Fixes #5265.
2017-11-16 09:45:08 -08:00
Eeshan Garg e85d0747d4 tools/test-help-documentation: Verify api/ docs. 2017-11-15 10:12:59 -08:00
rht f397e15128 tools: Text-wrap long lines exceeding 110. 2017-11-10 16:24:09 -08:00
rht 9c7d5812ce refactor: Remove six.moves.urllib.parse import. 2017-11-07 10:51:44 -08:00
neiljp (Neil Pilgrim) abc0d5e79b mypy: Add type parameter for Set in check_help_documentation.py. 2017-11-04 19:47:44 -07:00
rht c4fcff7178 refactor: Replace super(.*self) with Python 3-specific super().
We change all the instances except for the `test_helpers.py`
TimeTrackingCursor monkey-patching, which actually needs to specify
the base class.
2017-10-30 14:30:25 -07:00
rht e296841447 tools/documentation_crawler: Use python 3 syntax for typing. 2017-10-26 21:58:22 -07:00
derAnfaenger df948c7ea5 Remove useless shebang lines. 2017-10-11 12:52:36 -07:00
rht f15bdce90e tools: Remove print_function.
Tweaked by tabbott to exclude the linter libraries.
2017-09-29 15:44:56 -07:00
rht e455284bb4 requirements: Remove pathlib2.
`pathlib2` is a backport of pathlib to Python 2.x. Ni!
This dependency can be removed since:
- Zulip 1.6.0+git has been on Python 3 nowadays.
  Ekki-ekki-ekki-ptang-zoom-boing.
- As stated in ticket #6211, having this removed, prevents the need to
  have lockfiles for each of 3.4, 3.5, (and maybe 3.6).

This fixes #6211.
2017-08-23 10:00:21 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Vaida Plankyte 0ed4e22e54 docs: Fix lint/whitespace errors in GCI docs. 2017-06-22 09:30:28 -04:00
Aditya Bansal 69f9353ecf pep8: Add compliance with rule E261 to check_help_documentation.py. 2017-05-07 23:21:50 -07:00
Tim Abbott 3a1912e829 mypy: Fix missing documentation_crawler annotation. 2017-04-06 15:33:41 -07:00
Tim Abbott 10e9c3bb84 documentation_crawler: Add exclude list.
This works around the issue that Google calendar returns errors on
HTTP HEAD requests.
2017-04-06 15:26:27 -07:00
Rishi Gupta 2bbfdeeb7b Fix more errors caught by mypy 0.501.
Another set of relatively easy to review changes.
2017-03-03 14:15:38 -08:00
Rishi Gupta 28d3af0965 Fix several new errors caught by mypy 0.501.
Clear out a bunch of easy to review errors, so we can focus on the more
complicated ones.
2017-03-03 14:12:52 -08:00
K.Kanakhin aaf82ae090 documentation-crawler: Check images in help documentation.
This checks both that all images under static/images/help/ are used in
the help documentation, and also that none of the image tags are broken.

- Improve documentation spiders and crawler with spider error state.

Fixes #3070.
2017-01-12 10:02:32 -08:00
K.Kanakhin 589ae27190 documentation-crawler: Exclude privacy page from documentation checking 2016-12-19 08:31:39 -08:00
K.Kanakhin fa68a678cc help-documentation-tool: Add tools for checking user help documentation.
- Add spider for documentation crawler, which processes user help
  documentation on test development server.
- Add tool script for help documentation checking launching.

Fixes #2639
2016-12-16 09:04:04 -08:00
K.Kanakhin 2cbc7bb2c1 documentation-crawler: Create common spider for documentation checking.
Fixes #2639
2016-12-16 09:04:04 -08:00
K.Kanakhin b248a2c33e docs-test: Check only http 404 error for external links.
- Raise exception only for http 404 error rensponse, for other
  http error codes used logging to notify devs about wrong links.
2016-11-01 21:45:41 -07:00
K.Kanakhin 63f0e04621 docs-test: Exclude remote servers internal response errors.
Sometimes remote servers are unavailable or on maintance
what is the cause of broken back-end tests. Such response errors
is excluded from raised exception cases and just notify about
wrong link in test log.
2016-10-26 01:00:33 -07:00
K.Kanakhin 6b31fcd63a docs-test: Add scrapy crawler to check documentation not having broken links.
- Add scrapy project with rewrited 'crawl' command and spider to check documentation.
  Command was rewrited due to return exit status by the exception existing,
  it returns exit code 0 if exception happens inside spider in standard behavour.

- Add scrapy requirements to for dev environment. It was added to twisted requirements list.

Fixes #1492
2016-10-25 11:35:16 -07:00