There is only one PostgreSQL database; the "appdb" is irrelevant.
Also use "postgresql," as it is the name of the software, whereas
"postgres" the name of the binary and colloquial name. This is minor
cleanup, but enabled by the other renames in the previous commit.
This moves the puppet configuration closer to the "roles and profiles
method"[1] which is suggested for organizing puppet classes. Notably,
here it makes clear which classes are meant to be able to stand alone
as deployments.
Shims are left behind at the previous names, for compatibility with
existing `zulip.conf` files when upgrading.
[1] https://puppet.com/docs/pe/2019.8/the_roles_and_profiles_method
There was likely more dependency complexity prior to 97766102df, but
there is now no reason to require that consumers explicitly include
zulip::apt_repository.
Installing an updated linux kernel package, as can happen during the
`apt dist-upgrade` done by the installer, can cause grub to pop up a
prompt to update its configuration file. In an unattended headless
configuration, this will stop the installation.
Explicitly configure apt to be non-interactive, and prefer the newest
configuration, during the install.
We used to send occupy/vacate events when
either the first person entered a stream
or the last person exited.
It appears that our two main apps have never
looked at these events. Instead, it's
generally the case that clients handle
events related to stream creation/deactivation
and subscribe/unsubscribe.
Note that we removed the apply_events code
related to these events. This doesn't affect
the webapp, because the webapp doesn't care
about the "streams" field in do_events_register.
There is a theoretical situation where a
third party client could be the victim of
a race where the "streams" data includes
a stream where the last subscriber has left.
I suspect in most of those situations it
will be harmless, or possibly even helpful
to the extent that they'll learn about
streams that are in a "quasi" state where
they're activated but not occupied.
We could try to patch apply_event to
detect when subscriptions get added
or removed. Or we could just make the
"streams" piece of do_events_register
not care about occupy/vacate semantics.
I favor the latter, since it might
actually be what users what, and it will
also simplify the code and improve
performance.
I think it's important that the callers understand
that bulk_add_subscriptions assumes all streams
are being created within a single realm, so I make
it an explicit parameter.
This may be overkill--I would also be happy if we
just included the assertions from this commit.
In addition to being generally more correct, this works around a bug
in Node.js that causes webpack-dev-server to corrupt the terminal
state when exiting as a background process.
https://github.com/nodejs/node/issues/35536
Signed-off-by: Anders Kaseorg <anders@zulip.com>
It happens that whether you add a reaction or remove
a reaction, we send the exact same fields, just using
a different op code.
This sort of symmetry is actually kind of rare, as
usually "add" events have more fields, and "remove" events
might just send an id of something to remove.
Our openapi schema treats these as two seperate events,
so we are more consistent with it, and it helps our
schema-checking tooling for node fixtures, too.
Note that we now have to exempt the two events from
our openapi checks, due to the is_mirror_dummy field
in the deprecated user block. We can decide how to
handle this later--one possibility is to just add it
as an optional field on the event_schema side.
Note that we make the schema for profile_data
slightly more realistic, but it doesn't actually get
exercised by our current tests (apart from
making sure it's a dict), since we don't have
profile data for our test realm.
We also don't have the optional fields for bots,
since our tests don't exercise that, nor
delivery_email.
So we exempt realm_user_add_event from openapi
checks for now.
When we try to match the openapi specs better, we
will probably want to add a few tests to test_events.
Obviously getting good coverage for adding users
would be nice for all these scenarios:
* delivery_email matters
* bots
* realm has profile fields
This is a prep commit for supporting "presence"
events, where the key of the dictionary is some
arbitrary string like "website" but the value
of the dictionary is another dictionary itself
with keys that are more like variable names.
This also forces us to create TupleType.
We exempt this from the openapi check,
since we haven't figured out how to model
tuples in openapi with the same precision
as event_schema (and it may be impossible).
Long term we just want to stop dealing in
tuples, of course.
StringDict is a data type for representing dictionaries where
all keys and values are strings. Add this data type to data_types.py
and edit other files so that this data type is put to use and tested.
(slightly tweaked by @showell to remove a comment and shorten
a var name now that we have a proper data type)
We also make our schema in event_schema reflect this,
which in turn makes us match the already accurate
openapi spec, so we no longer need to exempt four
types of events from our sanity checks.
We might want to rename the tool to something more
general now, since we are really reconciling three
things:
- node fixtures
- event_schema checkers for test_events
- openapi specs
The way we compare python and openapi schemas is
as follows:
- first convert openapi schemas to be build
from DictType, ListType, etc. with from_opeapi
- do a diff on the schemas
Most of the new code is just having the FooType
family of classes serialize themselves with schema().
Defining types with an object hierarchy
of type classes will allow us to build
functionality that was impossible (or
really janky) with the validators.py
approach of composing functions.
Most of the changes to event_schema.py
were automated search/replaces.
This patch doesn't really yet take
advantage of the new FooType classes,
but we will use it soon to audit our
openapi specs.
SimpleBar 6.0.0-beta.2 through -beta.6 are built with ES6 syntax (I
assume inadvertently: https://github.com/Grsmto/simplebar/issues/523),
and its latest tag has moved back to 5.2.1 anyway.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
user_profile will be None for web_public_guests here. Hence, for
settings (of which most be inaccessible by web public guest),
which require a user_profile, we either set an empty value for
them or set them to a default value. This will help render
the frontend or extend support to our clients without breaking
a lot of code.
Tweaked by tabbott to add many comments.
The reason higher expected_time_to_clear_backlog were allowed for queues
during "bursts" was, in simpler terms, because those queues to which
this happens, intrinsically have a higher acceptable "time until cleared"
for new events. E.g. digests_email, where it's completely fine to take a
long time to send them out after putting in the queue. And that's
already configurable without a normal/burst distinction.
Thanks to this we can remove a bunch of overly complicated, and
ultimately useless, logic.
The race condition is described in the comment block removed by this
commit. This leaves room for another, remaining race condition
that should be virtually impossible, but nevertheless it seems
worthwhile to have it documented in the code, so we put a new comment
describing it.
As a final note, this is not a new race condition,
it was hypothetically possible with the old code as well.
We need this information in the frontend to:
* Display the 'view in playground' option for locally echoed messages.
* When we add a UI settings for realm admins to configure their
playground choices, we'll need to use these canonicalized aliases
for displaying the option.
Hence, this tweaks the tool which generates pygments_data.json to contain
the data we need.
Bumping major PROVISION_VERSION since folks need to provision in both
directions.
Tests amended.
css-loader@4 broke @import statements referencing files with
extensions other than .css, unless those @import statements are
compiled away by another loader. Upstream is more interested in
arguing that such @import statements are semantically incorrect than
applying the one line fix.
https://github.com/webpack-contrib/css-loader/issues/1164
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This was a broken abstraction that returned to its caller within
multiple forked processes on exceptions, and encouraged ignoring the
error code (as all of its callers did).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
The specification says “any sibling elements of a $ref are ignored”,
so their presence, although not invalid, indicates a mistake. yamole
incorrectly merges these siblings into the referenced object, but we
should not rely on this nonstandard behavior.
https://swagger.io/docs/specification/using-ref/#sibling
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Any exception is an "unexpected event", which means talking about
having an "unexpected event logger" or "unexpected event exception" is
confusing. As the error message in `exceptions.py` already explains,
this is about an _unsupported_ event type.
This also switches the path that these exceptions are written to,
accordingly.
8e10ab282a moved UnexpectedWebhookEventType into
`zerver.lib.exceptions`, but left the import into
`zserver.lib.webhooks.common` so that webhooks could continue to
import the exception from there.
This clutters things and adds complexity; there is no compelling
reason that the exception's source of truth should not move alongside
all other exceptions.
There are three functional side effects:
• Correct an insignificant but mathematically offensive bias toward
repeated characters in generate_api_key introduced in commit
47b4283c4b4c70ecde4d3c8de871c90ee2506d87; its entropy is increased
from 190.52864 bits to 190.53428 bits.
• Use the base32 alphabet in confirmation.models.generate_key; its
entropy is reduced from 124.07820 bits to the documented 120 bits, but
now it uses 1 syscall instead of 24.
• Use the base32 alphabet in get_bigbluebutton_url; its entropy is
reduced from 51.69925 bits to 50 bits, but now it uses 1 syscall
instead of 10.
(The base32 alphabet is A-Z 2-7. We could probably replace all of
these with plain secrets.token_urlsafe, since I expect most callers
can handle the full urlsafe_b64 alphabet A-Z a-z 0-9 - _ without
problems.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
It was broken by commit aaedec1fdb which
moved it to tools/i18n without adjusting its relative path references,
and it contains a sketchy injectable os.system call that I’d like to
remove.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
In 6653e19e3a we added
the convenient line to tell folks about the coverage
report. But if we failed coverage checks, we didn't
show the link.
Arguably we should just always show this, even if
tests fail, but that can also be potentially confusing.
The code to run single files was added
in c15695e514,
and it's just kinda strange code.
We already do a lot of file logic in Python
to check for line-coverage, so it's easier
to just have all the logic in Python.
This adds a new feature--you can now specify
the actual file:
./tools/test-js-with-node frontend_tests/node_tests/people.js
(This is helpful if you just want to use
shell autocomplete.)
Another minor change is that if you specify
individual files, we won't sort them. This is
important when you're trying to hunt down test
leaks.
Finally, we have a nicer message if we can't find
the file.
nyc was added in 29f04511c0
All the stuff after "&&" was actually passed to
node, because we didn't use shell=True, so the
"nyc report" command didn't run, and the ugly
finder.js code just skipped over all the final tokens.
We lost the war against top level configuration files many moons ago.
This is what developers and tools expect. And it seems to be required
for eslint-import-resolver-webpack (there’s ostensibly a {"config":
"tools/webpack.config.ts"} option, but it doesn’t work correctly:
https://github.com/benmosher/eslint-plugin-import/issues/1861).
Signed-off-by: Anders Kaseorg <anders@zulip.com>
This fixes the regression introduced in the pervious
commit to regain the 100% line coverage in `user_pill.js`
as well as `stream_pill.js`.
The new `stream_edit.js` mainly tests for:
* The stream related queries of the typeahead in `user_pill.js`
* The "Add subscribers" event handlers.
* The event handler which displays the settings for a stream.
We update the pills typeahead logic to also include
stream results and pass the "stream" key in `opts`
to enable this option for the Add subscriber form.
This commit implements the feature of adding all the
subscribers of another stream in the "Add subscribers"
UI, with the help of a new "stream_pill.js` file.
We temporarily add `user_pill.js` to the EXEMPT_FILES
list as typeahead will be set up in `stream_edit.js`
file which does not have any dedicated tests file.
Work towards #15186.
We merge bootstrap-responsive.css into bootsrap.css since that is
how bootstrap distributes it from this version onwards.
bootstrap.js has a lot of changes to it which completely breaks
our typeaheads and popovers, so we will have to override these
plugins with our version of these plugins. In future versions
of bootstrap when we use npm, we can just choose not to
import them.
The node package allow use to control xvfb apt package in puppeteer
tests. This help us create a fake display so we can run puppeteer in
headful (headless: false) mode, which is required to use the chrome
extension desktop capture API.
Now that all casper tests have been migrated to
puppeteer, there's no need for having casper
related things.
Removed the casperjs package and removed/replaced
casper in few places with puppeteer.
Only removed few of them which I'm confident
about. Also didn't make any changes in docs
as it would be easier to remove them while
adding puppeteer docs.
This rule is a bit marginal, in that we've only seen this mistake
once, but it is really subtle and took a while for translators to
notice it, so seems worth linting for anyway.
This commit adds automatic detection of extra output (other than
printed by testing library or tools) in stderr and stdout by code under
test test-backend when it is run with flag --ban-console-output.
It also prints the test that produced the extra console output.
Fixes: #1587.
There are file sharing issues with the macOS 10.15.6 and
vagrant. var/remote_cache_prefix was an empty file when using
VirtualBox and Docker on macOS.
Using parallels as a provider for vagrant fixes the issue.
Use --watch-poll which makes webpack to recompile
automatically on file changes, since inotify is not
working here too.
These weren’t wrong since orjson.JSONDecodeError subclasses
json.JSONDecodeError which subclasses ValueError, but the more
specific ones express the intention more clearly.
(ujson raised ValueError directly, as did json in Python 2.)
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Since our Webpack config passes pre-minified JS files to
script-loader, they can’t be used as modules. Use the normal
unminified version, letting Webpack minify it and give us source maps.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
The hash keys were missing hash for package.json and yarn.lock
because they were not present since we don't do a full checkout
in this job. We fix this by sending over those files and generating
hashes from them.
I usally verify these cache keys by clicking the Restore <cache>
step dropdown menu and then clicking the Run ... dropdown menu again
to see the generated hash.
ES and TypeScript modules are strict by default and don’t need this
directive. ESLint will remind us to add it to new CommonJS files and
remove it from ES and TypeScript modules.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Prior to commit eb4a2b9d4e the center
area of the navbar was based on a structure that appended crumbs or
"tabs" as <li>s, forming a tab_bar and a tab_list.
However, in eb4a2b9d4e we apply a new
style and structure to the navbar which lets go of the convention of
tabs. Hence, we'd like to purge the tab_bar and tab_list labels from
our code base.
We purged tab_list in 1267caf5009118875f47fdafe312880af08024e1.
This commit purges tab_bar, it includes:
- A blanket search and replace of tab_bar with message_view_header.
- Splitting a single line comment in
tab_bar.js / message_view_header.js.
- The renaming of tab_bar.js to message_view_header.js.
- The renaming of tab_bar.hbs to message_view_header.hbs.
- A blanket search and replace of tab_data with
message_view_header_data.
- Replacing the single occurrence of tabbar with message_view_header
(it was within a comment.)
There were a lots of flakes in CI recently because typeahead didn't
appear when Enter was pressed and real emails are not accepted as
valid inputs. To fix this we wait for typeahead to appear and then
click that instead of Enter. We also use delay option to type the
email (100ms delay between keypresses) since without we'd also get
flakes.
Re-enable puppeteer test in CI after this fix too.
😛 should be the most general version, which is the one
with open eyes. Other apps do the same and it also means that :P, which
is converted to 😛 is rendered like the emoticon.
Fixes#15970.
This is used rarely enough that it’s easier to document how to use it
as a non-global than to document the horrifying things that might go
wrong as a global.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
Anders Kaseorg
Vishnu KS
Alex Vandiver
YashRE42
Mateusz Mandera
Aman Agrawal
Steve Howell
Tim Abbott
orientor
Sumanth V Rao
Anders Kaseorg
Dinesh
Ryan Rehman
Priyank Patel
Mohit Gupta
Gittenburg