This was a broken abstraction that returned to its caller within
multiple forked processes on exceptions, and encouraged ignoring the
error code (as all of its callers did).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The specification says “any sibling elements of a $ref are ignored”,
so their presence, although not invalid, indicates a mistake. yamole
incorrectly merges these siblings into the referenced object, but we
should not rely on this nonstandard behavior.
https://swagger.io/docs/specification/using-ref/#sibling
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Any exception is an "unexpected event", which means talking about
having an "unexpected event logger" or "unexpected event exception" is
confusing. As the error message in `exceptions.py` already explains,
this is about an _unsupported_ event type.
This also switches the path that these exceptions are written to,
accordingly.
8e10ab282a moved UnexpectedWebhookEventType into
`zerver.lib.exceptions`, but left the import into
`zserver.lib.webhooks.common` so that webhooks could continue to
import the exception from there.
This clutters things and adds complexity; there is no compelling
reason that the exception's source of truth should not move alongside
all other exceptions.
There are three functional side effects:
• Correct an insignificant but mathematically offensive bias toward
repeated characters in generate_api_key introduced in commit
47b4283c4b4c70ecde4d3c8de871c90ee2506d87; its entropy is increased
from 190.52864 bits to 190.53428 bits.
• Use the base32 alphabet in confirmation.models.generate_key; its
entropy is reduced from 124.07820 bits to the documented 120 bits, but
now it uses 1 syscall instead of 24.
• Use the base32 alphabet in get_bigbluebutton_url; its entropy is
reduced from 51.69925 bits to 50 bits, but now it uses 1 syscall
instead of 10.
(The base32 alphabet is A-Z 2-7. We could probably replace all of
these with plain secrets.token_urlsafe, since I expect most callers
can handle the full urlsafe_b64 alphabet A-Z a-z 0-9 - _ without
problems.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
It was broken by commit aaedec1fdb which
moved it to tools/i18n without adjusting its relative path references,
and it contains a sketchy injectable os.system call that I’d like to
remove.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
In 6653e19e3a we added
the convenient line to tell folks about the coverage
report. But if we failed coverage checks, we didn't
show the link.
Arguably we should just always show this, even if
tests fail, but that can also be potentially confusing.
The code to run single files was added
in c15695e514,
and it's just kinda strange code.
We already do a lot of file logic in Python
to check for line-coverage, so it's easier
to just have all the logic in Python.
This adds a new feature--you can now specify
the actual file:
./tools/test-js-with-node frontend_tests/node_tests/people.js
(This is helpful if you just want to use
shell autocomplete.)
Another minor change is that if you specify
individual files, we won't sort them. This is
important when you're trying to hunt down test
leaks.
Finally, we have a nicer message if we can't find
the file.
nyc was added in 29f04511c0
All the stuff after "&&" was actually passed to
node, because we didn't use shell=True, so the
"nyc report" command didn't run, and the ugly
finder.js code just skipped over all the final tokens.
We lost the war against top level configuration files many moons ago.
This is what developers and tools expect. And it seems to be required
for eslint-import-resolver-webpack (there’s ostensibly a {"config":
"tools/webpack.config.ts"} option, but it doesn’t work correctly:
https://github.com/benmosher/eslint-plugin-import/issues/1861).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This fixes the regression introduced in the pervious
commit to regain the 100% line coverage in `user_pill.js`
as well as `stream_pill.js`.
The new `stream_edit.js` mainly tests for:
* The stream related queries of the typeahead in `user_pill.js`
* The "Add subscribers" event handlers.
* The event handler which displays the settings for a stream.
We update the pills typeahead logic to also include
stream results and pass the "stream" key in `opts`
to enable this option for the Add subscriber form.
This commit implements the feature of adding all the
subscribers of another stream in the "Add subscribers"
UI, with the help of a new "stream_pill.js` file.
We temporarily add `user_pill.js` to the EXEMPT_FILES
list as typeahead will be set up in `stream_edit.js`
file which does not have any dedicated tests file.
Work towards #15186.
We merge bootstrap-responsive.css into bootsrap.css since that is
how bootstrap distributes it from this version onwards.
bootstrap.js has a lot of changes to it which completely breaks
our typeaheads and popovers, so we will have to override these
plugins with our version of these plugins. In future versions
of bootstrap when we use npm, we can just choose not to
import them.
The node package allow use to control xvfb apt package in puppeteer
tests. This help us create a fake display so we can run puppeteer in
headful (headless: false) mode, which is required to use the chrome
extension desktop capture API.
Now that all casper tests have been migrated to
puppeteer, there's no need for having casper
related things.
Removed the casperjs package and removed/replaced
casper in few places with puppeteer.
Only removed few of them which I'm confident
about. Also didn't make any changes in docs
as it would be easier to remove them while
adding puppeteer docs.
This rule is a bit marginal, in that we've only seen this mistake
once, but it is really subtle and took a while for translators to
notice it, so seems worth linting for anyway.
This commit adds automatic detection of extra output (other than
printed by testing library or tools) in stderr and stdout by code under
test test-backend when it is run with flag --ban-console-output.
It also prints the test that produced the extra console output.
Fixes: #1587.
There are file sharing issues with the macOS 10.15.6 and
vagrant. var/remote_cache_prefix was an empty file when using
VirtualBox and Docker on macOS.
Using parallels as a provider for vagrant fixes the issue.
Use --watch-poll which makes webpack to recompile
automatically on file changes, since inotify is not
working here too.
These weren’t wrong since orjson.JSONDecodeError subclasses
json.JSONDecodeError which subclasses ValueError, but the more
specific ones express the intention more clearly.
(ujson raised ValueError directly, as did json in Python 2.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Since our Webpack config passes pre-minified JS files to
script-loader, they can’t be used as modules. Use the normal
unminified version, letting Webpack minify it and give us source maps.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The hash keys were missing hash for package.json and yarn.lock
because they were not present since we don't do a full checkout
in this job. We fix this by sending over those files and generating
hashes from them.
I usally verify these cache keys by clicking the Restore <cache>
step dropdown menu and then clicking the Run ... dropdown menu again
to see the generated hash.
ES and TypeScript modules are strict by default and don’t need this
directive. ESLint will remind us to add it to new CommonJS files and
remove it from ES and TypeScript modules.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Prior to commit eb4a2b9d4e the center
area of the navbar was based on a structure that appended crumbs or
"tabs" as <li>s, forming a tab_bar and a tab_list.
However, in eb4a2b9d4e we apply a new
style and structure to the navbar which lets go of the convention of
tabs. Hence, we'd like to purge the tab_bar and tab_list labels from
our code base.
We purged tab_list in 1267caf5009118875f47fdafe312880af08024e1.
This commit purges tab_bar, it includes:
- A blanket search and replace of tab_bar with message_view_header.
- Splitting a single line comment in
tab_bar.js / message_view_header.js.
- The renaming of tab_bar.js to message_view_header.js.
- The renaming of tab_bar.hbs to message_view_header.hbs.
- A blanket search and replace of tab_data with
message_view_header_data.
- Replacing the single occurrence of tabbar with message_view_header
(it was within a comment.)
There were a lots of flakes in CI recently because typeahead didn't
appear when Enter was pressed and real emails are not accepted as
valid inputs. To fix this we wait for typeahead to appear and then
click that instead of Enter. We also use delay option to type the
email (100ms delay between keypresses) since without we'd also get
flakes.
Re-enable puppeteer test in CI after this fix too.
😛 should be the most general version, which is the one
with open eyes. Other apps do the same and it also means that :P, which
is converted to 😛 is rendered like the emoticon.
Fixes#15970.
This is used rarely enough that it’s easier to document how to use it
as a non-global than to document the horrifying things that might go
wrong as a global.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This makes sure out fixture data for node tests
is realistic, according to the schemas in
zerver/lib/event_schema.py.
Note that we are still in the process of extracting
schemas from test_events.py -> event_schema.py,
so the checks here are somewhat incomplete as of
now.
One nice thing is that the program will tell us
what checkers are missing, so this can motivate
us to move more checkers to event_schema.py.
I considered just making this happen as part of
tools/test-js-with-node, but it's convenient to
run by itself. Also, it currently requires
Django (although we could fix that), which makes
it just expensive enough that I wouldn't want
to always run it before the node tests.
The previous steps for standing up a new host were somewhat manual.
This further scripts the process, by using the AWS CLI to start the
instance, and pass it a "user data" script to provision itself upon
boot. This results in a hands-off provisioning process which
completes in 5min.
Additional settings are required for `~/.zulip-install-server.conf`.
It is not suited for all roles, as it assumes one instance type and
security group value. Additionally, not all of the post-provision
process is currently automated -- Nagios SSH key verification, for
instance, is still a manual step. There are also additional steps for
database or frontend servers. Regardless, this is a move toward
automated provisioning.
Including anon=1 in API requests will retrieve all contributors
of the repo. If there is no asscoiated GitHub account present for
the commits then the email and name of the author mentioned in
commit messages is returned.
Previously, we copied them to /tmp and from there we specified those
assets we copied in circleci config in presist_to_workspace step.
Copying it to a directory allows us to get rid of list in circleci
config and GitHub Actions's upload artifact (their version of
presist to workspace) doesn't allow us to specify indivivual files
so only is this cleaner but required.
As of commit 87e72ac8e2 (#15267), we
need to be an owner for some of the tested functionality, not just an
administrator.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Zulip converts :) to the 1F642 Unicode emoji and promotes the same emoji
in the popular section of the emoji picker.
Previously Zulip has labeled 1F642 as "slight smile". While that name
conforms to the Unicode standard (which describes the code point as
SLIGHTLY SMILING FACE), it didn't match our use case of the emoji.
If a user types :) or selects the first smile in the emoji picker they
probably mean to express a regular "smile" and not a "slight smile",
which raises the question why they are only smiling slightly.
This commit relabels 1F642 as 😄 and our previous 😄 263A as
:smiling_face:. Note that 263A looks different in our three supported
emoji sets, so it is not suited to be our "default smile".
This change does not require a migration since our emoji system stores
both unicode points and names and handles name changes transparently.
Prettier would do this anyway, but it’s separated out for a more
reviewable diff. Generated by ESLint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Prettier would do this anyway, but it’s separated out for a more
reviewable diff. Generated by ESLint.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
A few major themes here:
- We remove short_name from UserProfile
and add the appropriate migration.
- We remove short_name from various
cache-related lists of fields.
- We allow import tools to continue to
write short_name to their export files,
and then we simply ignore the field
at import time.
- We change functions like do_create_user,
create_user_profile, etc.
- We keep short_name in the /json/bots
API. (It actually gets turned into
an email.)
- We don't modify our LDAP code much
here.
Fixes#12868.
We now also include python version in the format
'major.minor.patchlevel', when generating hash for a
requirement file. This was necessary since packages tend to
break on different versions of python, so it is important to
track the version on which the venv was setup.
WARN: This commit will force all zulip venvs to be recreated.
success-http-headers-bionic.txt and success-http-headers-focal.txt
differ only in the nginx version so this substitution will allow
us to have single file for both of them. Also this change helps
to avoid CI failure if Nginx version is updated in the OS.
The installer does not adjust the node name if the rabbitmq already
exists, and the default node name bakes in the
`zulip-install-bionic-base` hostname. As such, the resulting LXC
image does not properly start rabbitmq.
Remove rabbitmq, allowing the installer to install and configure it
with a nodename of `zulip@localhost`. This also lets the installed
image be successfully copied and booted under a new hostname without
breaking rabbitmq.
Doing service memcached start instead of restart fixed an issue on
focal build in GitHub actions, where it exits with code 1 when it
is done twice.It is done first in Install Dependencies step and then
again in last step where we call tools/ci/setup-backend again which
runs provision.
Furthermore, I don't belive there is a technical reason we use
restart over start; rather I think it was just a random choice with
the intend to just start the services in CI. I traced the code back
to commit 1f2f497cab if it helps.
Looking at the source code of memcached, the step that's failing is:
start-stop-daemon --stop --quiet --oknodo --pidfile $PIDFILE
which is equivilent to: service memcached stop, we can rule out the
service memcache start since it works. Ideally, we do figure out and
solve the issue of why memcached fails when executing service
memcached stop but I am not equipped with debugging it. And this
workaround seems reasonable rather than a "hacky" solution.
For the relevant code in memcached see:
https://github.com/memcached/memcached/blob/master/scripts/memcached-init.
Finally, the change to the rest of services is for consistency.
According to @showell:
> All the slow decorators can die. That was a failed experiment of
> mine from 2014 days. I have meaning to kill them for a couple years
> now. I wrote this with the best of intentions, but I believe it's
> now just cruft. We never made a "fast" mode, for one. And we kept
> writing more and more slow tests, haha.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Rename rest of function names, classes and comments containing bugdoown
to markdown in test_markdown.py. Also change the refactored classes and
functions occurences in other files.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
Rename the file and all the refrences to file and module test_bugdown.py
to test_markdown.py.
This commit is part of series of commit that renames bugdown to markdown.
This commit is first of few commita which aim to change all the
bugdown references to markdown. This commits rename the files,
file path mentions and change the imports.
Variables and other references to bugdown will be renamed in susequent
commits.
Ubuntu 20.04 "focal" comes up to runlevel 5 several seconds before it
is able to successfully resolve hosts, causing `prepare-base` to fail
while fetching from the apt repositories.
Add an additional check to verify that outbound networking is running
before returning from `lxc-wait`.
As in the previous commit, we can no longer pre-install the wrong
version of postgres. Unfortunately, this leaves it out of the base
image and thus makes testing installs longer.
A generator that yields values without receiving or returning them is
an Iterator. Although every Iterator happens to be iterable, Iterable
is a confusing annotation for generators because a generator is only
iterable once.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Add arrow key navigation support for recent topics.
Simple jquery is used to allow navigation for filter buttons,
a grid system is used for navigation inside table.
Because of a security release that required a migration, there
are two migrations numbered 0261. To avoid breaking existing
installs renumbering the migrations, we skipped migration 0261
when running tools/renumber-migrations.
Previously, --interactive used to run tests from the start on a
repeated run triggered when tests failed and we decide to trigger
a re-run by pressing Enter key. Rerunning passed tests is of no interest.
It also used to run all tests in a loop even if all pass.
This commit fixes those both issues i.e it runs again from the
test that failed on pressing Enter and exits if all tests pass.
This fixes a bundle of issues where we were missing "" around
attributes coming from variables. In most cases, the variables were
integers or fixed constants from the Zulip codebase (E.g. the name of
an installed integration), but in at least one case it was
user-provided data that could potentially have security impact.
This adds support for a "spoiler" syntax in Zulip's markdown, which
can be used to hide content that one doesn't want to be immediately
visible without a click.
We use our own spoiler block syntax inspired by Zulip's existing quote
and math block markdown extensions, rather than requiring a token on
every line, as is present in some other markdown spoiler
implementations.
Fixes#5802.
Co-authored-by: Dylan Nugent <dylnuge@gmail.com>
The automated tests running in CircleCI don't actually use the `zulip`
db, so we can skip running migrations on it in some CircleCI shards to
save time.
NOTE: This only effects build jobs that run provision, except the
`production-build` job where we skip building the dbs altogether.
Migrations still run on `focal-backend` build job to ensure
we are testing all our development setup code.
These files can’t use f-strings yet because they need to run in Python
2 or Python 3.5.
Generated by pyupgrade.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The psycopg2.SQL API unfortunately doesn’t work with
django.db.migrations.RunSQL, so we need to take a detour into
PL/pgSQL for EXECUTE and format.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Generated by pyupgrade --py36-plus --keep-percent-format.
Now including %d, %i, %u, and multi-line strings.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Use read-only types (List ↦ Sequence, Dict ↦ Mapping, Set ↦
AbstractSet) to guard against accidental mutation of the default
value.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
There seems to have been a confusion between two different uses of the
word “optional”:
• An optional parameter may be omitted and replaced with a default
value.
• An Optional type has None as a possible value.
Sometimes an optional parameter has a default value of None, or None
is otherwise a meaningful value to provide, in which case it makes
sense for the optional parameter to have an Optional type. But in
other cases, optional parameters should not have Optional type. Fix
them.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
To take screenshots, we log in as Desdemonia who doesn't seem
to have permissions to the streams where we capture screenshots.
This commit fixes it by logging in as Iago.
Also added a waitForNavigation until DOMContentLoaded to
avoid flakes that could occur because of us going to another URL
before Iago gets logged in.
Fixes#2665.
Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.
Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start. I expect this change will increase pressure for us to split
those files, which isn't a bad thing.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Automatically generated by the following script, based on the output
of lint with flake8-comma:
import re
import sys
last_filename = None
last_row = None
lines = []
for msg in sys.stdin:
m = re.match(
r"\x1b\[35mflake8 \|\x1b\[0m \x1b\[1;31m(.+):(\d+):(\d+): (\w+)", msg
)
if m:
filename, row_str, col_str, err = m.groups()
row, col = int(row_str), int(col_str)
if filename == last_filename:
assert last_row != row
else:
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
with open(filename) as f:
lines = f.readlines()
last_filename = filename
last_row = row
line = lines[row - 1]
if err in ["C812", "C815"]:
lines[row - 1] = line[: col - 1] + "," + line[col - 1 :]
elif err in ["C819"]:
assert line[col - 2] == ","
lines[row - 1] = line[: col - 2] + line[col - 1 :].lstrip(" ")
if last_filename is not None:
with open(last_filename, "w") as f:
f.writelines(lines)
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The test_management_commands use in particular was causing pickling
errors when the test failed, because Python 3 filter returns an
iterator, not a list.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Run production suites on Ubuntu Focal.
Added separate success-http-headers files for Focal and Bionic.
Also excluded them from whitespace rules in lint.
memcached 1.5.22 in Ubuntu 20.04 has a bug where it looks for its SASL
configuration at /etc/sasl2/memcached.conf/memcached.conf instead of
/etc/sasl2/memcached.conf.
We already use a workaround for this while applying puppet configurations in
99e71f3786 but for docker builds we used
do memcached hack since we can not use systemd in docker containers.
This was written by Rishi for a very brief purpose a few years ago,
and it doesn't serve much purpose now other than to be a place we
update in code sweeps.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Since we use this option in our docker-zulip project also
so rather than using it as a test suite option we made it
more specific i.e. --build-release-tarball-only.
Some UTF-8 characters (★ for example) are not displayed correctly, with
fonts-liberation. Puppeteer recommends[1] installing fonts-freefont-ttf in
their docs on running Puppeteer in docker.
Provisioning forward is sufficient. There's no need to remove the
new font and replace it with the old font, I think.
[1]: https://github.com/puppeteer/puppeteer/blob/master/docs/troubleshooting.md#running-puppeteer-in-docker
We remove the "GROUP PMs" section that used
to be in the lower right sidebar.
Most of this is straightforward code removal.
A couple quick notes:
- The message fetching code now just
calls `huddle_data.process_loaded_messages`,
which we still need for search suggestions.
We removed `activity.process_loaded_messages`.
- The `huddle_data.process_loaded_messages`
function no longer needs to return `need_resize`.
- In `resize.js` we now just calculate
`res.buddy_list_wrapper_max_height` directly
from `usable_height`.
Fixes#14962
* codecov needs `.coverage` file in the pwd to upload coverage
results.
* concurrency='multiprocessing' forces `.coverage` file to have a
data_suffix, we explicity set it to "" for the file to have no suffix.
Filed issue as https://github.com/nedbat/coveragepy/issues/989
The bug this was working around does not affect our current toolchain,
as confirmed by grepping through the minified output.
(Also, this linter rule only matched calc(x + y) with two arguments
and we were already using calc($far_left_gutter_size + $left_col_size
+ 4px).)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
mock is just a backport of the standard library’s unittest.mock now.
The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).
The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Previously, we handled this code only in message_list_view.js.
Now we support rendering stream descriptions and some dynamic
elements can be rendered in them, so we extract this new module
and use it in both the places.
This commit adds python code to call javascript_examples.js in its
two supported modes. tools/test-api asserts that the example output
is as expected, whereas the API markdown extension is used to render
these examples in the docs.
Anders Kaseorg
Anders Kaseorg
Tim Abbott
Alex Vandiver
Dinesh
Steve Howell
Ryan Rehman
Aman Agrawal
Priyank Patel
Mohit Gupta
YashRE42
Gittenburg
Vishnu KS
arpit551
Aman Agrawal
Felix
Clara Dantas
Sara Gulotta
Sharif Naas
Puneeth Chaganti
sahil839
Rohitt Vashishtha