Commit Graph

263 Commits

Author SHA1 Message Date
Tim Abbott 3e6b90a433 analytics: Explode coverage checks for analytics/views. 2021-06-17 17:39:28 -07:00
Suyash Vardhan Mathur 001435a297 refactor: Rename and move app_filters.py.
Moved the code in zerver/templatetags/app_filters.py
to more intuitive location and name at
zerver/lib/templates.py.
2021-06-11 07:43:22 -07:00
Mateusz Mandera 1d9fb4f988 django: Upgrade Zulip to Django 3.2 LTS.
This is a straightforward upgrade in terms of changes needed.

Necessary changes were:
- Set `DEFAULT_AUTO_FIELD`
  https://docs.djangoproject.com/en/3.2/releases/3.2/#customizing-type-of-auto-created-primary-keys

- `The default_app_config application configuration variable is deprecated, due
  to the now automatic AppConfig discovery.`
  https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery

To handle this one, we can remove default_app_config from
zerver/__init__.py because it satisfies what release notes describe in
https://docs.djangoproject.com/en/3.2/releases/3.2/#automatic-appconfig-discovery:
"Most pluggable applications define an AppConfig subclass in an apps.py
submodule. Many define a default_app_config variable pointing to this
class in their __init__.py.  When the apps.py submodule exists and
defines a single AppConfig subclass, Django now uses that configuration
automatically, so you can remove default_app_config."

An important note is that rebuild-test-database needs to be run after
this upgrade in dev environment - if tests are run with test db that was
built on the previous version, they will fail due to a mysterious bug
(?), where changing attributes of a user and .save()ing after logging in
in the test via self.login_user, causes getting logged out - the next
requests via self.client_get etc. are unauthed for some reason,
unless self.login_user is called again. This behavior is no longer
exhibited upon rebuilding the test db - and I can't reproduce it in
production or dev db. So this can likely be reasonably dismissed as some
quirk of the test client system that won't be relevant in the future and
doesn't impact production.
2021-05-03 08:36:22 -07:00
Aman Agrawal 59c8f3ed92 ci: Use general terms for CircleCI.
GitHub Actions supports doing more than just CI,
and so in some contexts it's less obvious that we're
talking about just the CI if we refer to it instead of CircleCI.
2021-03-16 14:56:43 -07:00
m-e-l-u-h-a-n 911c5f19f1 tools: Rename force argument to skip-provision-check
This commit renames --force argument used with various tests to
--skip-provision-check. As a consequence of this name change all other
files that set --force option for the test commands have been updated.

This change is done in order to provide more clarity for using this
option for runnning tests.

This commit addresses issue #17455.
2021-03-03 09:17:49 -08:00
m-e-l-u-h-a-n ca515e5583 tools: Move registration of --force to test-scripts.py.
This commit moves --force option used with various tests to
test-scripts.py to have it alongside the logic that does provisioning
status assertion.
This is a step towards providing more clarity over use of this
argument with tests as asked in issue #17455.
2021-03-03 09:17:49 -08:00
Anders Kaseorg 6e4c3e41dc python: Normalize quotes with Black.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 11741543da python: Reformat with Black, except quotes.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Anders Kaseorg 5028c081cb python: Merge concatenated string literals that Black would uglify.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2021-02-12 13:11:19 -08:00
Aman Agrawal c685d36821 hipchat_import: Remove tool from codebase.
Remove functions and scripts used by HipChat import tool and
those which will no longer be required in future.
2020-12-23 08:28:49 -08:00
Anders Kaseorg bcb571c187 test-backend: Fix argparse error type.
ArgumentError takes two arguments.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-11-10 14:17:22 -08:00
Aman Agrawal 87cdd8433d home: Allow logged out user through home.
We allow user to load webapp without log-in. This is only
be enabled for developed purposes now. Production setups will
see no changes.
2020-11-02 17:07:12 -08:00
Aman Agrawal 3ec23e1a9d fetch_initial_state_data: Handle case of web public guests.
user_profile will be None for web_public_guests here.  Hence, for
settings (of which most be inaccessible by web public guest),
which require a user_profile, we either set an empty value for
them or set them to a default value. This will help render
the frontend or extend support to our clients without breaking
a lot of code.

Tweaked by tabbott to add many comments.
2020-09-23 12:11:15 -07:00
Anders Kaseorg a659542d84 parallel: Remove run_parallel.
This was a broken abstraction that returned to its caller within
multiple forked processes on exceptions, and encouraged ignoring the
error code (as all of its callers did).

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-09-14 16:22:23 -07:00
Anders Kaseorg f91d287447 python: Pre-fix a few spots for better Black formatting.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 17:51:09 -07:00
Anders Kaseorg fbfd4b399d python: Elide action="store" for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg b4597a8ca8 python: Elide default for store_{true,false} argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-03 16:17:14 -07:00
Anders Kaseorg a5dbab8fb0 python: Remove redundant dest for argparse arguments.
Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-09-02 11:04:10 -07:00
Mohit Gupta 3390a70bcd tests: Add extra console output detection in test-backend output.
This commit adds automatic detection of extra output (other than
printed by testing library or tools) in stderr and stdout by code under
test test-backend when it is run with flag --ban-console-output.
It also prints the test that produced the extra console output.

Fixes: #1587.
2020-08-27 11:39:53 -07:00
Anders Kaseorg 61d0417e75 python: Replace ujson with orjson.
Fixes #6507.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-08-11 10:55:12 -07:00
Alex Vandiver 39368cad3a tornado: Extract functions called from django into one module.
This makes clearer the separation of concerns.
2020-08-10 16:55:56 -07:00
Anders Kaseorg 92d0290dc5 tests: Remove slow tests detection.
According to @showell:

> All the slow decorators can die. That was a failed experiment of
> mine from 2014 days.  I have meaning to kill them for a couple years
> now.  I wrote this with the best of intentions, but I believe it's
> now just cruft.  We never made a "fast" mode, for one. And we kept
> writing more and more slow tests, haha.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-30 23:34:53 -07:00
Mohit Gupta 4224ac1b61 refactor: Rename remaining bugdown words to markdown in test_markdown.py.
Rename rest of function names, classes and comments containing bugdoown
to markdown in test_markdown.py. Also change the refactored classes and
functions occurences in other files.
This commit is part of series of commits aimed at renaming bugdown to
markdown.
2020-06-26 17:23:15 -07:00
Mohit Gupta 0578a918e6 refactor: Rename test_bugdown.py to test_markdown.py.
Rename the file and all the refrences to file and module test_bugdown.py
to test_markdown.py.
This commit is part of series of commit that renames bugdown to markdown.
2020-06-26 17:08:37 -07:00
Mohit Gupta 3f5fc13491 refactor: Rename zerver.lib.bugdown to zerver.lib.markdown .
This commit is first of few commita which aim to change all the
bugdown references to markdown. This commits rename the files,
file path mentions and change the imports.
Variables and other references to bugdown will be renamed in susequent
commits.
2020-06-26 17:08:37 -07:00
Anders Kaseorg 365fe0b3d5 python: Sort imports with isort.
Fixes #2665.

Regenerated by tabbott with `lint --fix` after a rebase and change in
parameters.

Note from tabbott: In a few cases, this converts technical debt in the
form of unsorted imports into different technical debt in the form of
our largest files having very long, ugly import sequences at the
start.  I expect this change will increase pressure for us to split
those files, which isn't a bad thing.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-11 16:45:32 -07:00
Anders Kaseorg 67e7a3631d python: Convert percent formatting to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-10 15:02:09 -07:00
Anders Kaseorg 8dd83228e7 python: Convert "".format to Python 3.6 f-strings.
Generated by pyupgrade --py36-plus --keep-percent-format, but with the
NamedTuple changes reverted (see commit
ba7906a3c6, #15132).

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-06-08 15:31:20 -07:00
Aman Agrawal c9bff71410 ci: Don't remove .coverage file & set empty suffix.
Fixes #14962
* codecov needs `.coverage` file in the pwd to upload coverage
results.
* concurrency='multiprocessing' forces `.coverage` file to have a
data_suffix, we explicity set it to "" for the file to have no suffix.
Filed issue as https://github.com/nedbat/coveragepy/issues/989
2020-05-27 17:54:54 -07:00
Anders Kaseorg 840cf4b885 requirements: Drop direct dependency on mock.
mock is just a backport of the standard library’s unittest.mock now.

The SAMLAuthBackendTest change is needed because
MagicMock.call_args.args wasn’t introduced until Python
3.8 (https://bugs.python.org/issue21269).

The PROVISION_VERSION bump is skipped because mock is still an
indirect dev requirement via moto.

Signed-off-by: Anders Kaseorg <anders@zulip.com>
2020-05-26 11:40:42 -07:00
Rohan Prashant 44a56a66d0 tests-backend: Bring stream_recipient.py to 100 percent coverage.
Fixes part of #7089.
2020-05-12 16:43:01 -07:00
Steve Howell 239474124e test-backend: Remove generate_fixtures option.
We remove the `generate_fixtures` option here mostly
for simplicity, but in particular to facilitate
an upcoming commit to simplify the job of
`generate-fixtures` (and remove its `--force` option).

The command line option here for `test-backend`
was really calling `generate_fixtures --force`,
which we're about to rename `tools/rebuild-test-database`.

The `test-backend` tools is already smart about catching
up on migrations, so we generally don't need to tell it
to repair the database.

And if the database does get corrupt, you can just do
it directly with `tools/rebuild-test-database`.

This eliminates the `use_force` flag in
`update_test_databases_if_required`, which was easy
to confuse with `rebuild_test_database`.

The other caller wasn't using `use_force`.
2020-04-22 09:24:42 -07:00
Mateusz Mandera fbc8325d0e test-backend: Remove rate_limiter from not_yet_fully_covered.
rate_limiter.py now has sufficient test coverage to remove from the
list of exclusions.

Tweaked by tabbott to handle @abstractmethod in a better way.
2020-04-15 11:20:37 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Vishnu KS a48845c8fb tests: Create new test mode for generating stripe fixtures. 2020-03-17 16:46:56 -07:00
Mateusz Mandera 782353f0e1 tests: Ensure coverage data files get cleaned up at the end.
Without calling cov.erase() the data file seems to persist and even
pollute future test runs if not removed. Registering an atexit handler
seems like a good, and reasonably clean way to ensure the cleanup
happens.
Fixes #13933.
2020-02-19 14:36:17 -08:00
Mateusz Mandera a25f00a69c populate_db: Extract some functions to server_initialization.py. 2020-01-29 17:26:45 -08:00
Tim Abbott d70e799466 bots: Remove FEEDBACK_BOT implementation.
This legacy cross-realm bot hasn't been used in several years, as far
as I know.  If we wanted to re-introduce it, I'd want to implement it
as an embedded bot using those common APIs, rather than the totally
custom hacky code used for it that involves unnecessary queue workers
and similar details.

Fixes #13533.
2020-01-25 22:41:39 -08:00
Mateusz Mandera 8dd95bd057 tests: Replace httpretty with responses.
responses is an module analogous to httpretty for mocking external
URLs, with a very similar interface (potentially cleaner in that it
makes use of context managers).

The most important (in the moment) problem with httpretty is that it
breaks the ability to use redis in parts of code where httpretty is
enabled.  From more research, the module in general has tendency to
have various troublesome bugs with breaking URLs that it shouldn't be
affecting, caused by it working at the socket interface layer.  While
those issues could be fixed, responses seems to be less buggy (based
on both third-party reports like ckan/ckan#4755 and our own experience
in removing workarounds for bugs in httpretty) and is more actively
maintained.
2020-01-22 11:56:15 -08:00
Anders Kaseorg ea6934c26d dependencies: Remove WebSockets system for sending messages.
Zulip has had a small use of WebSockets (specifically, for the code
path of sending messages, via the webapp only) since ~2013.  We
originally added this use of WebSockets in the hope that the latency
benefits of doing so would allow us to avoid implementing a markdown
local echo; they were not.  Further, HTTP/2 may have eliminated the
latency difference we hoped to exploit by using WebSockets in any
case.

While we’d originally imagined using WebSockets for other endpoints,
there was never a good justification for moving more components to the
WebSockets system.

This WebSockets code path had a lot of downsides/complexity,
including:

* The messy hack involving constructing an emulated request object to
  hook into doing Django requests.
* The `message_senders` queue processor system, which increases RAM
  needs and must be provisioned independently from the rest of the
  server).
* A duplicate check_send_receive_time Nagios test specific to
  WebSockets.
* The requirement for users to have their firewalls/NATs allow
  WebSocket connections, and a setting to disable them for networks
  where WebSockets don’t work.
* Dependencies on the SockJS family of libraries, which has at times
  been poorly maintained, and periodically throws random JavaScript
  exceptions in our production environments without a deep enough
  traceback to effectively investigate.
* A total of about 1600 lines of our code related to the feature.
* Increased load on the Tornado system, especially around a Zulip
  server restart, and especially for large installations like
  zulipchat.com, resulting in extra delay before messages can be sent
  again.

As detailed in
https://github.com/zulip/zulip/pull/12862#issuecomment-536152397, it
appears that removing WebSockets moderately increases the time it
takes for the `send_message` API query to return from the server, but
does not significantly change the time between when a message is sent
and when it is received by clients.  We don’t understand the reason
for that change (suggesting the possibility of a measurement error),
and even if it is a real change, we consider that potential small
latency regression to be acceptable.

If we later want WebSockets, we’ll likely want to just use Django
Channels.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-01-14 22:34:00 -08:00
Anders Kaseorg f077508c6d test-backend, run-casper: Remove proxy vars instead of setting to "".
Setting http_proxy to the empty string caused a spurious error from
xmlsec:

>>> import os
>>> os.environ["http_proxy"] = ""
>>> import xmlsec
error : Unknown IO error

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2019-10-28 15:47:45 -07:00
Vishnu KS 1585ad7bf4 mattermost: Add support for exporting DMs and huddles. 2019-10-10 16:37:03 -07:00
Tim Abbott 356e68e09e test-backend: Temporarily disable coverage requirement for mattermost.
The mattermost import tool has some temporary coverage regressions.
2019-09-18 12:11:58 -07:00
Wyatt Hoodes fc2574fdfd test_mattermost_importer: Add test coverage.
This is a rebase from a recent revert. A test
was failing non-deterministically relating to
`convert_channel_data` and use of an incorrect
`realm_id`.
2019-09-03 12:36:41 -07:00
Tim Abbott d0d16ed81c Revert "test_mattermost_importer: Fix missing coverage."
This reverts commit c48e53f587.

This new test code had nondeterministic failures; reverting so we can
debug without disruption.
2019-08-21 11:20:20 -07:00
Hemanth V. Alluri f280e9cf84 lib: Rename lib/api_test_helpers.py to openapi/python_examples.py
This will make the contained code easier to find.
2019-08-05 21:06:19 -07:00
Eeshan Garg 8e10ab282a webhooks: Log unexpected webhook events separately.
This change serves to declutter webhook-errors.log, which is
filled with too many UnexpectedWebhookEventType exceptions.

Keeping UnexpectedWebhookEventType in zerver/lib/webhooks/common.py
led to a cyclic import when we tried to import the exception in
zerver/decorators.py, so this commit also moves this exception to
another appropriate module. Note that our webhooks still import
this exception via zerver/lib/webhooks/common.py.
2019-07-22 18:20:53 -07:00
Wyatt Hoodes 0e3fddbc6e test_fixtures: Add logic for removing stale test directories.
Similarly to how stale database removal is handled, we add a check for
stale test run directories at the end of the `test-backend` script.
2019-07-12 17:42:18 -07:00
Wyatt Hoodes 16abd3d1e7 test-backend: Remove long-term clean up code.
With the refactored file structure approach, the clean up block at the
end of `test-backend` is no longer necessary.
2019-07-12 17:42:18 -07:00
Wyatt Hoodes b1fe1ef42b test-backend: Clean up leak data import files after test-suite run.
This is a simple, non-intrusive way of removing the bulk of the
clutter from `var/<uuid>/test-backend` after running `test-backend`.

Ideally, we'll replace this logic with proper tearDown methods.
2019-06-24 12:39:56 -07:00