In 2f547ea, the custom `has_error` logic in BaseDocumentationSpider
was removed in favor of checking whether any errors were logged.
One reference to that method was missed in UnusedImagesLinterSpider
and went unremarked as no images were added to the documentation
that were not used.
Replaces that reference with an error log.
Fixes a warning caused by using the deprecated `2.6` value
of the `REQUEST_FINGERPRINTER_IMPLEMENTATION` setting.
Upgrades `settings.py` to what would have been generated
by starting the documentation_crawler project using the
`startproject` command of scrapy v2.7.
Since /communities/#all was failing the spider test due to `#all`
not being present as an `id` on the page, we exclude it from
this check saying that it is not really a documentation page.
I have suppressed errors for github.com by adding an function to
exclude domains as well as urls; this is necessary because GitHub has
marked this tool's User-Agent as a blocker crawler.
I have also suppressed reoccurring url errors that do definitely exist.
Fixes#17928.
f82cc4ed06 started checking all
zulip/zulip GitHub links in CI. Instead, it should have checked only
zulip/zulip file and directory links since checking other
links require making requests to GitHub servers.
A generator that yields values without receiving or returning them is
an Iterator. Although every Iterator happens to be iterable, Iterable
is a confusing annotation for generators because a generator is only
iterable once.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Fixes#2665.
Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.
Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start. I expect this change will increase pressure for us to split
those files, which isn't a bad thing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Automatically generated by the following script, based on the output
of lint with flake8-comma:
import re
import sys
last_filename = None
last_row = None
lines = []
for msg in sys.stdin:
m = re.match(
r"\x1b\[35mflake8 \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
)
if m:
filename, row_str, col_str, err = m.groups()
row, col = int(row_str), int(col_str)
if filename == last_filename:
assert last_row != row
else:
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
with open(filename) as f:
lines = f.readlines()
last_filename = filename
last_row = row
line = lines[row - 1]
if err in ["C812", "C815"]:
lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
elif err in ["C819"]:
assert line[col - 2] == ","
lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This was written by Rishi for a very brief purpose a few years ago,
and it doesn't serve much purpose now other than to be a place we
update in code sweeps.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Tests for these links often result in rate-limiting from GitHub,
leading to the builds failing in Circle CI. We temporarily mark
github.com/zulip links as external to keep the builds passing.
Apparently, the CircleCI and Codecov links (and the Codecov badge)
weren't pointing specifically at master, so they'd sometimes show
state from the lastest push to a pull request, which isn't a
reasonable way to advertise whether the project's build is passing.
Previous cleanups (mostly the removals of Python __future__ imports)
were done in a way that introduced leading newlines. Delete leading
newlines from all files, except static/assets/zulip-emoji/NOTICE,
which is a verbatim copy of the Apache 2.0 license.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
For .start-button, Bootstrap carousel already supports <button
data-target> as a valid alternative to <button href>. For
.call-to-action, the margin is decreased to exactly offset the lack of
margin collapsing with display: inline-block. There should be no
visual change.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The VNU_IGNORE whitelist lets in some crazy-invalid preexisting HTML,
but hopefully this will stop the problem from getting much larger.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This way we inherit more of the upstream command’s behavior.
Importantly, this means we pass everything in `opts.spargs` to the
spider, not just `opts.spargs.skip_external`.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
This doesn’t seem to add any noise in the normal case, but if anything
shows up here we might want to see it.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
* Remove the custom has_error logic in favor of checking whether any
errors were logged, which gives us a much better chance at catching
unanticipated exceptions.
* Use our error_callback for the initial requests of start_urls too.
* Clean up mypy types.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
Spider raises exceptions when errors like FileNotFound
are detected. However, these did not set error state
before exiting causing spider to fail silently.
This patch sets the status causing exceptions to exit with
non-zero exit status.
Lauryn Menard
David Rosa
Aman Agrawal
Anders Kaseorg
Alex Vandiver
Eeshan Garg
Alya Abbott
Adam Birds
Vishnu KS
Anders Kaseorg
Tim Abbott
Rohitt Vashishtha
Tim Abbott
sumanthvrao
Anders Kaseorg