Commit Graph

114 Commits

Author SHA1 Message Date
Vishnu KS d4f5cea559 docs: Mark chat.zulip.org as an external link for testing.
The comment explains the background, but basically there's no purpose
in trying to verify this via CI.
2020-10-29 16:42:17 -07:00
Anders Kaseorg 72d6ff3c3b docs: Fix more capitalization issues.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-10-23 11:46:55 -07:00
Aman Agrawal bdc4721aee BaseDocumentationSpider: Don't crawl webapp. 2020-10-16 16:26:29 -07:00
Aman Agrawal 72b25553b2 BaseDocumentationSpider: Print correct link on error. 2020-10-16 16:23:33 -07:00
Anders Kaseorg f91d287447 python: Pre-fix a few spots for better Black formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg 3ffed617a2 mypy: Type simple generators as Iterator, not Iterable.
A generator that yields values without receiving or returning them is
an Iterator.  Although every Iterator happens to be iterable, Iterable
is a confusing annotation for generators because a generator is only
iterable once.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-23 11:29:54 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 69730a78cc python: Use trailing commas consistently.
Automatically generated by the following script, based on the output
of lint with flake8-comma:

import re
import sys

last_filename = None
last_row = None
lines = []

for msg in sys.stdin:
    m = re.match(
        r"\x1b\[35mflake8    \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
    )
    if m:
        filename, row_str, col_str, err = m.groups()
        row, col = int(row_str), int(col_str)

        if filename == last_filename:
            assert last_row != row
        else:
            if last_filename is not None:
                with open(last_filename, "w") as f:
                    f.writelines(lines)

            with open(filename) as f:
                lines = f.readlines()
            last_filename = filename
        last_row = row

        line = lines[row - 1]
        if err in ["C812", "C815"]:
            lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
        elif err in ["C819"]:
            assert line[col - 2] == ","
            lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")

if last_filename is not None:
    with open(last_filename, "w") as f:
        f.writelines(lines)

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-06-11 16:04:12 -07:00
Tim Abbott 7b8ba5ebd9 docs: Update most remaining references to zulipchat.com.
In some cases, the cleanest tweak is to replace references to the
domain with Zulip Cloud, the product.
2020-06-08 18:10:45 -07:00
Tim Abbott 95be7dcbab portico: Remove mystery hunt landing page.
This was written by Rishi for a very brief purpose a few years ago,
and it doesn't serve much purpose now other than to be a place we
update in code sweeps.
2020-06-08 18:10:45 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Anders Kaseorg f8339f019d python: Convert assignment type annotations to Python 3.6 style.
Commit split by tabbott; this has changes to scripts/, tools/, and
puppet/.

scripts/lib/hash_reqs.py, scripts/lib/setup_venv.py,
scripts/lib/zulip_tools.py, and tools/lib/provision.py are excluded so
tools/provision still gives the right error message on Ubuntu 16.04
with Python 3.5.

Generated by com2ann, with whitespace fixes and various manual fixes
for runtime issues:

-shebang_rules: List[Rule] = [
+shebang_rules: List["Rule"] = [

-trailing_whitespace_rule: Rule = {
+trailing_whitespace_rule: "Rule" = {

-whitespace_rules: List[Rule] = [
+whitespace_rules: List["Rule"] = [

-comma_whitespace_rule: List[Rule] = [
+comma_whitespace_rule: List["Rule"] = [

-prose_style_rules: List[Rule] = [
+prose_style_rules: List["Rule"] = [

-html_rules: List[Rule] = whitespace_rules + prose_style_rules + [
+html_rules: List["Rule"] = whitespace_rules + prose_style_rules + [

-    target_port: int = None
+    target_port: int

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-24 13:06:54 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Rohitt Vashishtha b25e02aa14 tests: Treat github.com/zulip links as external.
Tests for these links often result in rate-limiting from GitHub,
leading to the builds failing in Circle CI. We temporarily mark
github.com/zulip links as external to keep the builds passing.
2020-03-19 09:27:40 -04:00
Tim Abbott fa26475fcf docs: Deal with a few broken links. 2019-10-15 14:03:53 -07:00
Tim Abbott 23c48764b5 README: Improve links to coverage/CI to point to master.
Apparently, the CircleCI and Codecov links (and the Codecov badge)
weren't pointing specifically at master, so they'd sometimes show
state from the lastest push to a pull request, which isn't a
reasonable way to advertise whether the project's build is passing.
2019-10-01 15:31:55 -07:00
Anders Kaseorg becef760bf cleanup: Delete leading newlines.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines.  Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-08-06 23:29:11 -07:00
Anders Kaseorg 82828bdba4 HTML validation: Remove invalid <button href> attribute.
For .start-button, Bootstrap carousel already supports <button
data-target> as a valid alternative to <button href>.  For
.call-to-action, the margin is decreased to exactly offset the lack of
margin collapsing with display: inline-block.  There should be no
visual change.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-23 16:16:22 -07:00
Anders Kaseorg 586aaddcc0 documentation_crawler: Stop treating all portico URLs as external.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-07-18 11:56:15 -07:00
Anders Kaseorg a1cf77ca54 test-help-documentation: Bind vnu.jar to 127.0.0.1.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-28 18:18:29 -07:00
Anders Kaseorg 2d50dcf7cc test-help-documentation: Validate HTML with vnu.jar.
The VNU_IGNORE whitelist lets in some crazy-invalid preexisting HTML,
but hopefully this will stop the problem from getting much larger.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-27 14:53:21 -07:00
Anders Kaseorg bac63f5a3e documentation_crawler: Reimplement crawl_with_status so it wraps crawl.
This way we inherit more of the upstream command’s behavior.
Importantly, this means we pass everything in `opts.spargs` to the
spider, not just `opts.spargs.skip_external`.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-06-24 23:20:17 -07:00
Tim Abbott efe5e53118 docs: Update list of links to not check in tests. 2019-06-17 12:24:22 -07:00
Anders Kaseorg d9f12cfb49 documentation_crawler: They’re called fragments, not permalinks.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg 4f28b88746 documentation_crawler: Set a 15 second download timeout.
By default it just hangs for as long as the server keeps the
connection open.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg c59747360a documentation_crawler: Set the default log level to warning.
This doesn’t seem to add any noise in the normal case, but if anything
shows up here we might want to see it.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
Anders Kaseorg 2f547eaee4 documentation_crawler: Improve error handling.
* Remove the custom has_error logic in favor of checking whether any
  errors were logged, which gives us a much better chance at catching
  unanticipated exceptions.
* Use our error_callback for the initial requests of start_urls too.
* Clean up mypy types.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-05-30 10:39:47 -07:00
sumanthvrao c5de6d0e18 spiders: Set error status for exceptions generated by spider.
Spider raises exceptions when errors like FileNotFound
are detected. However, these did not set error state
before exiting causing spider to fail silently.

This patch sets the status causing exceptions to exit with
non-zero exit status.
2019-04-08 17:04:29 -07:00
Anders Kaseorg ee8ff4df66 tools: Remove unused imports.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2019-02-02 17:10:31 -08:00
Tim Abbott 5a8ad84381 test-documentation: Exclude the 500ing Citizen Code of Conduct.
We might want to eventually remove this, but I'm holding out hope that
it'll be fixed.
2018-12-29 15:30:19 -08:00
Vishnu Ks 6f5b1903c4 docs: Update code-reviewing docs to reflect CI changes. 2018-12-29 15:28:26 -08:00
Sumanth V Rao 776b981dde test-help-documentation: Skips checking external links.
This option causes test-help-documentation to verify internal links only.
This prevents documentation from flaking in CI due to links from
external websites having failed in the documentation.

Based on conversation:
https://chat.zulip.org/#narrow/stream/43-automated-testing/subject/quora.20404/near/674997
2018-12-29 20:15:30 +05:30
Sumanth V Rao e1f33e0f6b test-documentation: Add --skip-external-links option.
This option causes test-documentation to only verify internal links
that we control and are important to be correct.  This prevents
test-documentation from flaking in CI due to issues with the dozens of
third-party blocks that we link to from various parts of our
documentation.

Tweaked by tabbott for comment clarity, and to also include
github.com/zulip links.

Fixes #10942.
2018-12-05 12:39:59 -08:00
Tim Abbott 723fb60476 docs: Fix up some broken links output spam. 2018-11-28 09:49:39 -08:00
Rishi Gupta 69e9a8f196 user docs: Add a few portico URLs to documentation crawler test. 2018-11-01 11:59:50 -07:00
Tim Abbott d9887759b4 plans: Fix broken link from FAQ to import docs and add tests. 2018-10-31 11:44:21 -07:00
Anders Kaseorg 037f696d26 Enable pycodestyle W605 (invalid escape sequence).
The only changes visible at the AST level, checked using
https://github.com/asottile/astpretty, are

zerver/lib/test_fixtures.py:
'\x1b\\[(1|0)m' ↦ '\\x1b\\[(1|0)m'
'\\[[X| ]\\] (\\d+_.+)\n' ↦ '\\[[X| ]\\] (\\d+_.+)\\n'

which is fine because re treats '\\x1b' and '\\n' the same way as
'\x1b' and '\n'.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2018-07-03 16:54:46 +02:00
Eeshan Garg ca5ea20ab7 doc tests: Add portico pages to tools/test-help-documentation.
Fixes #9117.
2018-04-21 21:51:32 -07:00
Tim Abbott b0b134cb4c help: Clean up settings_html and subscriptions_html.
After some thinking, I don't think there's any actual value to doing
the ../ style relative links here, whereas there is actual harm from
the links being slightly broken in the current model.  We fix this by
just using /#settings as the URL.

Fixes #8978.
2018-04-05 14:48:26 -07:00
Tim Abbott ed5b374ffa help docs: Fix a sorta buggy URL test. 2018-04-05 13:07:34 -07:00
David Rosa Tamsen 7072fa5b37 docs: Reorganize developer docs to improve navigation.
This commit helps reduce clutter on the navigation sidebar.
Creates new directories and moves relevant files into them.
Modifies index.rst, symlinks, and image paths accordingly.

This commit also enables expandable/collapsible navigation items,
renames files in docs/development and docs/production,
modifies /tools/test-documentation so that it overrides a theme setting,
Also updates links to other docs, file paths in the codebase that point
to developer documents, and files that should be excluded from lint tests.

Note that this commit does not update direct links to
zulip.readthedocs.io in the codebase; those will be resolved in an
upcoming follow-up commit (it'll be easier to verify all the links
once this is merged and ReadTheDocs is updated).

Fixes #5265.
2017-11-16 09:45:08 -08:00
Eeshan Garg e85d0747d4 tools/test-help-documentation: Verify api/ docs. 2017-11-15 10:12:59 -08:00
rht f397e15128 tools: Text-wrap long lines exceeding 110. 2017-11-10 16:24:09 -08:00
rht 9c7d5812ce refactor: Remove six.moves.urllib.parse import. 2017-11-07 10:51:44 -08:00
neiljp (Neil Pilgrim) abc0d5e79b mypy: Add type parameter for Set in check_help_documentation.py. 2017-11-04 19:47:44 -07:00
rht c4fcff7178 refactor: Replace super(.*self) with Python 3-specific super().
We change all the instances except for the `test_helpers.py`
TimeTrackingCursor monkey-patching, which actually needs to specify
the base class.
2017-10-30 14:30:25 -07:00
rht e296841447 tools/documentation_crawler: Use python 3 syntax for typing. 2017-10-26 21:58:22 -07:00
derAnfaenger df948c7ea5 Remove useless shebang lines. 2017-10-11 12:52:36 -07:00
rht f15bdce90e tools: Remove print_function.
Tweaked by tabbott to exclude the linter libraries.
2017-09-29 15:44:56 -07:00
rht e455284bb4 requirements: Remove pathlib2.
`pathlib2` is a backport of pathlib to Python 2.x. Ni!
This dependency can be removed since:
- Zulip 1.6.0+git has been on Python 3 nowadays.
  Ekki-ekki-ekki-ptang-zoom-boing.
- As stated in ticket #6211, having this removed, prevents the need to
  have lockfiles for each of 3.4, 3.5, (and maybe 3.6).

This fixes #6211.
2017-08-23 10:00:21 -07:00
Greg Price a099e698e2 py3: Switch almost all shebang lines to use `python3`.
This causes `upgrade-zulip-from-git`, as well as a no-option run of
`tools/build-release-tarball`, to produce a Zulip install running
Python 3, rather than Python 2.  In particular this means that the
virtualenv we create, in which all application code runs, is Python 3.

One shebang line, on `zulip-ec2-configure-interfaces`, explicitly
keeps Python 2, and at least one external ops script, `wal-e`, also
still runs on Python 2.  See discussion on the respective previous
commits that made those explicit.  There may also be some other
third-party scripts we use, outside of this source tree and running
outside our virtualenv, that still run on Python 2.
2017-08-16 17:54:43 -07:00
Vaida Plankyte 0ed4e22e54 docs: Fix lint/whitespace errors in GCI docs. 2017-06-22 09:30:28 -04:00
Aditya Bansal 69f9353ecf pep8: Add compliance with rule E261 to check_help_documentation.py. 2017-05-07 23:21:50 -07:00
Tim Abbott 3a1912e829 mypy: Fix missing documentation_crawler annotation. 2017-04-06 15:33:41 -07:00
Tim Abbott 10e9c3bb84 documentation_crawler: Add exclude list.
This works around the issue that Google calendar returns errors on
HTTP HEAD requests.
2017-04-06 15:26:27 -07:00
Rishi Gupta 2bbfdeeb7b Fix more errors caught by mypy 0.501.
Another set of relatively easy to review changes.
2017-03-03 14:15:38 -08:00
Rishi Gupta 28d3af0965 Fix several new errors caught by mypy 0.501.
Clear out a bunch of easy to review errors, so we can focus on the more
complicated ones.
2017-03-03 14:12:52 -08:00
K.Kanakhin aaf82ae090 documentation-crawler: Check images in help documentation.
This checks both that all images under static/images/help/ are used in
the help documentation, and also that none of the image tags are broken.

- Improve documentation spiders and crawler with spider error state.

Fixes #3070.
2017-01-12 10:02:32 -08:00
K.Kanakhin 589ae27190 documentation-crawler: Exclude privacy page from documentation checking 2016-12-19 08:31:39 -08:00
K.Kanakhin fa68a678cc help-documentation-tool: Add tools for checking user help documentation.
- Add spider for documentation crawler, which processes user help
  documentation on test development server.
- Add tool script for help documentation checking launching.

Fixes #2639
2016-12-16 09:04:04 -08:00
K.Kanakhin 2cbc7bb2c1 documentation-crawler: Create common spider for documentation checking.
Fixes #2639
2016-12-16 09:04:04 -08:00
K.Kanakhin b248a2c33e docs-test: Check only http 404 error for external links.
- Raise exception only for http 404 error rensponse, for other
  http error codes used logging to notify devs about wrong links.
2016-11-01 21:45:41 -07:00
K.Kanakhin 63f0e04621 docs-test: Exclude remote servers internal response errors.
Sometimes remote servers are unavailable or on maintance
what is the cause of broken back-end tests. Such response errors
is excluded from raised exception cases and just notify about
wrong link in test log.
2016-10-26 01:00:33 -07:00
K.Kanakhin 6b31fcd63a docs-test: Add scrapy crawler to check documentation not having broken links.
- Add scrapy project with rewrited 'crawl' command and spider to check documentation.
  Command was rewrited due to return exit status by the exception existing,
  it returns exit code 0 if exception happens inside spider in standard behavour.

- Add scrapy requirements to for dev environment. It was added to twisted requirements list.

Fixes #1492
2016-10-25 11:35:16 -07:00