Commit Graph

3784 Commits

Author SHA1 Message Date
Steve Howell 900fe02471 tools: Remove replacer.
This seems to be a complicated wrapper around `sed`.
It's not used anywhere or documented anywhere.  I
assume it was used in some code sweep or something.

See cec45b0ae5
2020-04-21 14:12:57 -07:00
Steve Howell 735f098ff5 tools: Remove get-handlebars-vars.
I built this in 2013 to help me quickly build
some sample data for our very early node tests,
I believe.

Ever since then it's just been changed for various
code sweeps.  Also, if we wanted to resurrect this
idea for some reason, we now have the template
parser that was written since then.
2020-04-21 14:12:57 -07:00
Steve Howell 3dd1a9211c tools: Remove html-grep.
I built this in August 2016, but I haven't used it
since, and all the more recent commits are just for
things like mypy sweeps.
2020-04-21 14:12:57 -07:00
Steve Howell 5bf43b77df tools: Remove django-template-graph.
This tool was last modified in 2016, has no
documentation, and requires `dot` to run, so
I think we can remove it to reduce some clutter
in the tools directory.
2020-04-21 14:12:57 -07:00
Steve Howell 2488438d90 tools: Convert fix-unused-css to look at zulip.scss.
This tool is hackish and incomplete, but it's still
worth looking at if somebody wants to hunt down
obsolete CSS.  There are probably better ways to
tackle this problem, so we should eventually just
remove this tool, but it's pretty low maintenance.

Current output looks like this:

    $ ./tools/find-unused-css
    actions_hovered
    actions_link
    bookend_tr
    btn-skip
    company-name
    flatpickr-months
    label_for_text
    loading_more_messages_indicator_box
    loading_more_messages_indicator_box_container
    logoimage
    messages-collapse
    messages-expand
    numInputWrapper
    page_loading_indicator_box
    page_loading_indicator_box_container
    skinny-user-gravatar
    sp-input
    summary_colorblock
    summary_row
    summary_row_private_message
    tutorial-done-button
2020-04-21 14:12:57 -07:00
Vishnu KS 87fdef2472 team: Don't remove duplicate commits inside the retry loop.
Else for each retry the duplicate commits would be removed
again and again from the contributor's zulip/zulip commits.
This is a bug in the original commit
6bed6ccdcf that added the
functionality to remove duplicate commits.
2020-04-21 13:41:10 -07:00
Puneeth Chaganti 7a5033852d provision: Add dependencies for Chromium used by latest puppeteer.
a51902a870 upgraded the puppeteer version to
3.0.0, which upgraded the Chromium version being used[1]. This version of
Chromium requires a couple of extra libraries to be installed.

[1]: https://github.com/puppeteer/puppeteer/releases/tag/v3.0.0
2020-04-20 23:25:45 -07:00
Steve Howell 51f74a7fd8 provision: Manage digests more rigorously.
We now have two functions related to digests
for processes:

    is_digest_obsolete
    write_digest_file

In most cases we now **wait** to write the
digest file until after we've successfully
run a process with its new inputs.

In one place, for database migrations, we
continue to write the digest optimistically.
We'll want to fix this, but it requires a
little more code cleanup.

Here is the typical sequence of events:

    NEVER RUN -
        is_digest_obsolete returns True
        quickly (we don't compute a hash)

        write_digest_file does a write (duh)

    AFTER NO CHANGES -
        is_digest_obsolete returns False
        after reading one file for old
        hash and multiple files to compute
        hash

        most callers skip write_digest_file

        (no files are changed)

    AFTER SOME CHANGES -
        is_digest_obsolete returns False
        after doing full checks

        most callers call write_digest_file
        *after* running a process
2020-04-20 15:06:47 -07:00
Steve Howell 2bd836bc4c provision: Promote pygments import to module scope.
There's no real reason to do the lazy import any
more, as we use this unconditionally inside `main`
(indirectly), and `provision_inner` runs after we
have set up the venv.
2020-04-20 15:06:47 -07:00
Steve Howell e66bd6a7a4 provision: Put hash_name argument first (minor). 2020-04-20 15:06:47 -07:00
Steve Howell ca49f38619 provision: Extract helpers for paths to hash.
I make these all functions for consistency,
and in particular I want to continue to avoid
`glob.glob` calls until we are actually
computing hashes.

This is mostly a prep to allow us to do
hashing in two separate places:

    - check hashes
    - update hashes

We would only update hashes **after** running
processes anew.

For `provision_inner` I considered using a
class to put the three path-related helpers
into a mini namespace, but it felt too heavy.

It wouldn't be completely implausible here
to extract something like a JSON config
file that has a list of globs for each
process that we do path-hashing for, but I
want to clean up other stuff first.
2020-04-20 15:06:47 -07:00
Steve Howell 4822f8d7d6 refactor: Add Database.template_status.
This is mostly a pure code move from
template_database_status().
2020-04-20 15:06:47 -07:00
Steve Howell 0ea4f727d4 refactor: Change params to template_database_status(). 2020-04-20 15:06:47 -07:00
Steve Howell 5c5d85cf19 test databases: Add Database.run_db_migrations().
We can reduce some code duplication by having this
on the class.
2020-04-20 15:06:47 -07:00
Puneeth Chaganti e55d967f6e generate_zulip_bots_static_files: Lazy import cairosvg.
Importing cairosvg in fails in production, because `libgtk-3-dev` is not
available. This commit moves the import for `cairosvg` into the function
where it is used. This code will soon be moved into a separate script that
will not be run on production.
2020-04-20 09:14:08 -07:00
Anders Kaseorg 438a545477 gitlint-rules: Fix bogus type annotations.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-19 13:36:58 -07:00
Aman Agrawal b13beb5ea8 circleci: Remove Xenial build as we are deprecating Xenial.
Docs updated to reflect the change.
2020-04-19 11:38:49 -07:00
Anders Kaseorg 5901e7ba7e python: Convert function type annotations to Python 3 style.
Generated by com2ann (slightly patched to avoid also converting
assignment type annotations, which require Python 3.6), followed by
some manual whitespace adjustment, and six fixes for runtime issues:

-    def __init__(self, token: Token, parent: Optional[Node]) -> None:
+    def __init__(self, token: Token, parent: "Optional[Node]") -> None:

-def main(options: argparse.Namespace) -> NoReturn:
+def main(options: argparse.Namespace) -> "NoReturn":

-def fetch_request(url: str, callback: Any, **kwargs: Any) -> Generator[Callable[..., Any], Any, None]:
+def fetch_request(url: str, callback: Any, **kwargs: Any) -> "Generator[Callable[..., Any], Any, None]":

-def assert_server_running(server: subprocess.Popen[bytes], log_file: Optional[str]) -> None:
+def assert_server_running(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> None:

-def server_is_up(server: subprocess.Popen[bytes], log_file: Optional[str]) -> bool:
+def server_is_up(server: "subprocess.Popen[bytes]", log_file: Optional[str]) -> bool:

-    method_kwarg_pairs: List[FuncKwargPair],
+    method_kwarg_pairs: "List[FuncKwargPair]",

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 20:42:48 -07:00
Anders Kaseorg d3c55c166e requirements: Upgrade mypy from 0.761 to 0.770.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Anders Kaseorg 21c2a4cc42 update-locked-requirements: Generate prod locks from dev locks.
This guarantees that we don’t accidentally upgrade one without the
other, which could happen for example due to different third-party
version constraints between the two.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-18 13:09:51 -07:00
Steve Howell 7eb6d32d59 provision: Let build_emoji build its own cache.
We no longer need to maintain duplicate code
related to where we set up the emoji
cache directory.

And we no longer need two extra steps for
people doing advanced (i.e. manual) setup.

There was no clear benefit to having provision
build the cache directory for `build_emoji`,
when it was easy to make `build_emoji` more
self-sufficient.  The `build_emoji` tool
was already importing the library that has
`run_as_root`, and it was already responsible
for 99% of the create-directory kind of tasks.

(We always call `build_emoji` unconditionally from
`provision`, so there's no rationale in terms
of avoiding startup time or something.)

ASIDE:

Its not completely clear to me why we need
to put this directory in "/srv", instead of
somewhere more local (like we already do for
Travis), but maybe it's just to be like
its siblings in "/srv":

    node_modules
    yarn.lock
    zulip-emoji-cache
    zulip-npm-cache
    zulip-py3-venv
    zulip-thumbor-venv
    zulip-venv-cache
    zulip-yarn

I guess the caches that we keep in var are
dev-only, although I think some of what's under
`zulip-emoji-cache` is also dev-only in nature?

    ./var/webpack-cache
    ./var/mypy-cache

In `docs/subsystems/emoji.md` we say this:

```
The `build_emoji` tool generates the set of files under
`static/generated/emoji` (or really, it generates the
`/srv/zulip-emoji-cache/<sha1>/emoji` tree, and
`static/generated/emoji` is a symlink to that tree;we do this in
order to cache old versions to make provisioning and production
deployments super fast in the common case that we haven't changed the
emoji tooling). [...]
```

I don't really understand that rationale for the development
case, since `static/generated` is as much ignored by `git` as
'/srv' is, without the complications of needing `sudo` to create it.

And in production, I'm not sure how much time we're really saving,
as it takes me about 1.4s to fully rebuild the cache in dev, not to
mention we're taking on upgrade risk by sharing files between versions.
2020-04-17 09:53:26 -07:00
Steve Howell bf3decfd0c build_emoji: Inline get_success_stamp.
So, `source_emoji_dump` is not the greatest variable
name, but at least we now define it relative to its
parent instead of the `.success-stamp` file.  (And
then `success_stamp` is just another join.)
2020-04-17 09:45:59 -07:00
Steve Howell aae28e41d0 provision: Extract create_var_directories().
This fixes the pattern of
SIX_DIR_PATH_CONSTANTS_IN_ALL_CAPS
just to create a few subdirectories.
2020-04-17 09:45:59 -07:00
Steve Howell bc62796276 provision: Check for compiled templates.
If the directory `templates/zerver/emails/compiled/`
is missing, then we need to run `inline_email_css`
again.

This can happen if somebody gets overzealous about
cleaning untracked files.
2020-04-17 09:45:59 -07:00
Steve Howell ec416a315f provision: Extract need_to_run_inline_email_css.
This extraction also prevents an unnecessary
call to `glob.glob` when `is_force` is `True`.
2020-04-17 09:45:59 -07:00
Steve Howell 66234850ba provision: Extract need_to_run_build_pygments_data.
This is more encapsulated and more efficient.

In the cases where `is_force` is `True` or
`pygments_data.json` is missing, we now avoid
the unnecessary step of importing `pygments`, at
least up front.

(Of course, we probably import that once we generate
the artifacts.)
2020-04-17 09:45:59 -07:00
Steve Howell 796351ed9e provision: Detect missing language_name_map.json.
If somebody is having issues with provision, it's
plausible they'll do something like `git clean -fX`
to clean up old artifacts of earlier provision runs,
as part of debugging things.

We defend against this by detecting the most obvious
symptom as cheaply as possible.
2020-04-17 09:45:59 -07:00
Steve Howell 765d40b34c provision: Extract need_to_run_compilemessages.
This is slightly more than a code move, as we
now skip some unnecessary file I/O when
`is_force` is `True`.
2020-04-17 09:45:59 -07:00
Steve Howell 067196c508 provision: Simplify `is_force` codepaths.
I remove `is_force` from `file_or_package_hash_updated`
and modernize its mypy annotations.

If `is_force` is `True`, we just now run the thing
we want to force-run without having to call
`file_or_package_hash_updated` to expensively
and riskily return `True`.

Another nice outcome of this change is that if
`file_or_package_hash_updated` returns `True`,
you can know that the file or package has
indeed been updated.

For the case of `build_pygments_data` we also
skip an `os.path.exists` check when `is_force`
is `True`.

We will short-circuit more logic in the next
few commits, as well as cleaning up some of
the long/wrapper lines in the `if` statements.
2020-04-17 09:45:59 -07:00
Steve Howell f98843d197 provision: Add build_emoji prefix to message.
This is similar to what we do in generate_secrets,
and it makes the output from a no-op provision
a bit more consistent.
2020-04-17 09:45:59 -07:00
Steve Howell 478b50dde3 provision: Improve message for configure-rabbitmq.
We change the message for skipping RabbitMQ
configuration to match nearby messages:

    No need to run `tools/setup/build_pygments_data`.
    No need to run `scripts/setup/inline_email_css.py`.
    No need to run `scripts/setup/configure-rabbitmq.
    No need to regenerate the dev DB.
    No need to regenerate the test DB.
    No need to run `manage.py compilemessages`.
2020-04-17 09:45:59 -07:00
Puneeth Chaganti 6b3358b14f generate-integration-docs-screenshot: Allow sending custom headers.
This commit makes it convenient to specify additional headers to send in the
integration webhook request from the command line.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 84e411c543 generate-integration-docs-screenshot: Fix crash on ignored fixtures.
When using a fixture that is correctly handled by our webhooks, but don't
cause a notification message to be sent, the tool shouldn't crash.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 9248b16e86 message-screenshot: Create image directory if it doesn't exist. 2020-04-17 09:41:55 -07:00
Puneeth Chaganti cee240fb74 generate-integration-docs-screenshot: Raise ArgumentTypeError.
When an invalid fixture is passed as an argument to the tool, raise an
`ArgumentTypeError`, instead of a `ValueError`.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti ae754887eb generate-integration-docs-screenshot: Message to run dev server.
If the dev server is not running, when this tool is run, we print a clear
error message to start the dev server first, and then run this tool.
2020-04-17 09:41:55 -07:00
Puneeth Chaganti 874e473fc4 generate-integration-docs-screenshot: Refactor code into functions. 2020-04-17 09:41:55 -07:00
Puneeth Chaganti 4719226d0f generate-integration-docs-screenshot: Ensure puppeteer is installed. 2020-04-17 09:41:55 -07:00
Puneeth Chaganti dd6966b9af test-js-with-puppeteer: Extract code to prepare for puppeteer run. 2020-04-17 09:41:55 -07:00
Puneeth Chaganti 572e188b36 tools: Extract code to find js test files to test_script. 2020-04-17 09:41:55 -07:00
Puneeth Chaganti 26e199035d test_script: Use Python3 style type hints. 2020-04-17 09:41:55 -07:00
Tim Abbott 561ded5e59 production: Fix generating bot static files in production.
For upgrade-zulip-from-git to work, we need to be able to run
update-prod-static on production systems, which means provision code
like this cairosvg logic needs to be there for now.
2020-04-17 09:25:48 -07:00
Puneeth Chaganti 464cbcd9c1 tools: Capture incoming bot message screenshot using puppeteer. 2020-04-16 19:25:13 -07:00
Puneeth Chaganti 4d2ce607c9 tools: Add script to trigger webhook notification using fixtures.
When creating a webhook integration or creating a new one, it is a pain to
create or update the screenshots in the documentation. This commit adds a
tool that can trigger a sample notification for the webhook using a fixture,
that is likely already written for the tests.

Currently, the developer needs to take a screenshot manually, but this could
be automated using puppeteer or something like that.

Also, the tool does not support webhooks with basic auth, and only supports
webhooks that use json fixtures. These can be fixed in subsequent commits.
2020-04-16 19:25:13 -07:00
Puneeth Chaganti 4c142b778a generate_zulip_bots_static_files: Move imports outside the function. 2020-04-16 19:25:13 -07:00
Mateusz Mandera fbc8325d0e test-backend: Remove rate_limiter from not_yet_fully_covered.
rate_limiter.py now has sufficient test coverage to remove from the
list of exclusions.

Tweaked by tabbott to handle @abstractmethod in a better way.
2020-04-15 11:20:37 -07:00
wowol fcf4c9639e emails: Add tests for sending custom emails.
This requires configuring the linter to allow using "subject" in these
files.
2020-04-14 10:50:43 -07:00
Tim Abbott 4f23f13c55 fetch-contributor-data: Fix running in production.
The previous logic attempted to check for a venv, rather than putting
itself into the virtualenv automatically.
2020-04-12 12:14:07 -07:00
Aman Agrawal cf4349d1c1 tools: Add script to start database services for WSL.
It needs to be run everytime WSL is restarted.
2020-04-11 12:05:33 -07:00
Steve Howell b8552e9524 test-js-with-node: Show correct coverage link in color.
We figure out the dev host using the same logic as
dev_settings.py, so that we don't use wrong things
like 127.0.0.1 for droplet users.

And we display the link in cyan.
2020-04-11 07:05:04 -04:00
Steve Howell f28a1a4c6c test-js-with-node: Remove stray comments.
When I introduced EXEMPT_FILES, I forgot
to delete these comments, and they became
totally nonsensical after sorting the list
of files.
2020-04-11 07:05:04 -04:00
wowol 71dfb85607 emails: Allow usage of word subject in send_email.py. 2020-04-10 15:53:35 -07:00
Tim Abbott 59937ec9fa scripts: Rename inline-email-css to inline_email_css.py.
This is a preparatory step for making it a module that we can import
and call from other code.
2020-04-10 15:29:47 -07:00
Anders Kaseorg d8fce9417b run-dev: Automatically set EXTERNAL_HOST for droplet dev servers.
As of commit 99242138a7 (#14530), this
is required when visiting a droplet dev server remotely.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-10 16:50:13 -04:00
Anders Kaseorg 99242138a7 static: Serve webpack bundles from the root domain.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-10 00:48:02 -07:00
Anders Kaseorg c734bbd95d python: Modernize legacy Python 2 syntax with pyupgrade.
Generated by `pyupgrade --py3-plus --keep-percent-format` on all our
Python code except `zthumbor` and `zulip-ec2-configure-interfaces`,
followed by manual indentation fixes.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-09 16:43:22 -07:00
Rohitt Vashishtha 3f6541b306 bugdown: Add 'none' as alias for no syntax highlighting in codeblocks.
This is a precursor to #14404.
2020-04-09 16:02:02 -07:00
Mateusz Mandera 122d0bca83 check-rabbitmq-queue: Add a simple algorithm to analyze queue stats.
This new algorithm is designed to avoid monitoring paging when a queue
simply has bursty behavior.
2020-04-09 13:41:01 -07:00
Vishnu KS 31a5119892 tools: Back off after request failure in fetch-contributor-data.
If a request fails the tool sleeps for some time before making
further requests. The sleep time is a random number between
0 and 2^failures capped at 64 seconds. More details about the
algorithm can be found at https://chat.zulip.org/#narrow/stream/
92-learning/topic/exponential.20backoff.20--.20with.20jitter
2020-04-08 12:53:05 -07:00
Vishnu KS 449f7e2d4b team: Generate team page data using cron job.
This eliminates the contributors data as a possible source of
flakiness when installing Zulip from Git.

Fixes #14351.
2020-04-08 12:52:31 -07:00
Vishnu KS 8415a1472a tools: Rename update-authors-json to fetch-contributor-data. 2020-04-08 12:40:00 -07:00
Vishnu KS abad5365fa tools: Move duplicate_commits.json file to tools directory.
This is a prep commit for generating /team page data
using cron job. zerver/tests directory is not present in
production installation. So moving the file from the directory
tests to tools.
2020-04-07 16:06:29 -07:00
Greg Price 4d17abee82 tools: Use require_clean_work_tree everywhere.
As described in the commit that added this function, this fixes one
quite annoying bug and one at least in-principle bug:

 * On Windows, the simple version (lacking `git update-index
   --refresh`) routinely gives false positives, making the tools
   that rely on it basically unusable.

 * If you have uncommitted changes in the index but manage to have
   the worktree nevevertheless match HEAD, the simple version will
   give a false negative and we'd blow away those changes.
2020-04-06 17:52:10 -07:00
Greg Price daf357aaab tools: Give `require_clean_work_tree` our nice UI touches.
Notably, print the details of what's modified.
2020-04-06 17:52:10 -07:00
Greg Price 43ca39c87d tools: Modernize require_clean_work_tree a bit.
Use `local`.  Also `set -u`-compatible `${2-}`, and normalize
formatting of `if .. then`.
2020-04-06 17:52:10 -07:00
Greg Price 73acca76ba tools: Add a require_clean_work_tree function, from Git upstream.
This is verbatim from Git upstream, at an older version.  (The one
change since then is to add localization for the messages like "You
have unstaged changes" -- which complicates the code, is important and
worth it for Git itself, but for our tools we can do without.)

This function will replace our use of `git diff-index --quiet HEAD`
in several scripts.  The key differences in behavior are:

 * The `git update-index --refresh`.  Without this, on Windows
   apparently `git diff-index` routinely (but not all the time!)
   reports that tons of files have changed.  See report:
     https://chat.zulip.org/#narrow/stream/9-issues/topic/.2E.2Ftools.2Ffetch-pull-request.20issue/near/834435

 * Instead of one command comparing the worktree to HEAD, we
   separately compare the worktree to the index and the index to
   HEAD, and abort if either diff is nonempty.  This one is obvious,
   but rather an edge case (it matters only if you've managed to
   make the worktree and HEAD agree while the index has some
   changes), and the extra code is annoying if written out in every
   script that needs it.  But that's what a subroutine is for. :-)

We'll make a few tweaks before actually switching to use this.
2020-04-06 17:52:10 -07:00
Greg Price fc0b99cd62 tools: Apply `set -x` to just the interesting parts of *-pull-request.
The Git commands we're invoking to do the real work are useful to
print, for transparency to see what's happening and that there's no
magic here.

The boring shell stuff like `remote=${2:-"upstream"}` is not so
helpful, and nor is the rather arcane and in any case read-only
command `git diff-index --quiet HEAD`.  Those only add noise that
obscures the interesting parts.  So, move the `set -x` down to when
we're done with the boring preparatory stuff and ready to perform
the commands that do the work.
2020-04-06 17:52:10 -07:00
Anders Kaseorg 15d68c40dd nginx: Set X-XSS-Protection: 1; mode=block.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-05 16:13:53 -07:00
Anders Kaseorg 79c215626e nginx: Set X-Content-Type-Options: nosniff globally.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-05 16:13:53 -07:00
Anders Kaseorg bcc040b01c test-locked-requirements: Cache only one copy of each valid hash.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-05 13:45:25 -07:00
Anders Kaseorg bc6862b949 test-locked-requirements: Check all locked requirements files.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-05 13:45:25 -07:00
Steve Howell da6508a7af presence tests: Test for offline/reload use cases.
This gets us to 100% line coverage.
2020-04-03 11:42:59 -07:00
brendon f65e6d0d94 sgrep: Install syntactic code search tool as an external linter.
Add sgrep (sgrep.dev) to tooling and include simple rule as
proof of concept. Included rule detects use of old django render
function.

Also added a rule that looks for if-else statements where both
code paths are identical.
2020-04-01 15:08:34 -07:00
Anders Kaseorg 68cfcd6446 CVE-2020-9444: Prevent reverse tabnabbing attacks.
While we could fix this issue by changing the markdown processor,
doing so is not a robust solution, because even a momentary bug in the
markdown processor could allow cached messages that do not follow our
security policy.

This change ensures that even if our markdown processor has bugs that
result in rendered content that does not properly follow our policy of
using rel="noopener noreferrer" on links, we'll still do something
reasonable.

Co-authored-by: Tim Abbott <tabbott@zulipchat.com>
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-04-01 14:01:45 -07:00
arpit551 a2dd1fad12 provision: switch Focal to use PGroonga from PPA.
Since Groonga packages for Ubuntu 20.04 Focal are now available
in their ppa so stopped building pgroonga from source.
2020-03-31 15:02:31 -07:00
Anders Kaseorg 9d75eb0ad5 install-shellcheck: Check the tarball’s SHA-256 hash.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-31 15:01:54 -07:00
Steve Howell 6b86d9055f tests: Only wait up to 90s for the server. 2020-03-29 12:17:16 -07:00
Steve Howell cef74e1fdd tests: Sleep a bit more between server-is-up checks.
A watched pot never boils.

If we get server startup time down enough where 400ms
becomes a significant chunk, we can re-tweak this.
2020-03-29 12:17:16 -07:00
Steve Howell 7e4d542df4 tests: Use zulipdev.com for server_is_up() check.
Folks can have issues connecting to Casper
as zulipdev.com when they are not connected to
the internet or just have a bad connection, since
the DNS record is on the internet.  Folks can
work around this by just creating an /etc/hosts
entry for zulipdev.com, but people don't always
know.

This fix moves the symptom slightly earlier in
the process--we don't advertise that the server
is "up" if you can't actually connect to it as
"zulipdev.com".
2020-03-29 12:17:16 -07:00
Stefan Weil d2fa058cc1
text: Fix some typos (most of them found and fixed by codespell).
Signed-off-by: Stefan Weil <sw@weilnetz.de>
2020-03-27 17:25:56 -07:00
Anders Kaseorg 43996106d8 requirements: Get transifex-client from apt.
In the past it has blocked Python library security updates with overly
strict version bounds, and we don’t use it as a library, only as a
binary.

Skip the PROVISION_VERSION bump because we can use the tx binary from
either location.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:39:04 -07:00
Anders Kaseorg 006a69c912 ci: Disable test-run-dev.
This test doesn’t catch many bugs and is known to be flaky.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:38:19 -07:00
Anders Kaseorg a6624f04db Revert "run-dev: Run process_queue with DJANGO_AUTORELOAD_ENV."
This reverts commit 36a8e61e67 (#13934).

The Django 2.2 autoreloader works by forking into a child process that
exits with status 3 when a file changes, and a parent process that
restarts the child when it exits with status 3.  Setting this
environment variable had the effect of pretending we were already the
child process, without a parent process to restart it.  Therefore,
changing any code used by the queue processor caused it to exit rather
than restart.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:38:19 -07:00
Anders Kaseorg 7ff9b22500 docs: Convert many http URLs to https.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-26 21:35:32 -07:00
arpit551 b5c2202b3b provision: add dependency for building pgroonga from source.
clang-9 and llvm-9-dev are added for building pgroonga from source.
2020-03-25 16:09:59 -07:00
arpit551 f23a435fcb circleci: Add base image for Ubuntu Focal.
Added base image for Focal and new job in config.yml
to run backend and frontend tests in circleci for
python3.8 using Focal docker image.
2020-03-25 16:09:59 -07:00
arpit551 b9c8c185bc circleci: Removed python-dev from dockerfile.template.
python-dev will be depreciated in Focal but can be used as python2-dev
so removed it from common dockerfile.template and added it
as an extra package in .circleci/config.yml.
2020-03-25 16:09:59 -07:00
arpit551 0c821424cd lint: Run whitespace linter against .yml files.
Lint now checks for .yml files with whitespace_rules.
2020-03-24 15:31:59 -07:00
Anders Kaseorg 39f9abeb3f python: Convert json.loads(f.read()) to json.load(f).
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-03-24 10:46:32 -07:00
Tim Abbott 3d0b720543 provision: Fix missing apt update for installing new dependencies.
Previously, we only did apt updates when our sources.list files or
keys changed, which could result in provisioning errors for
development systems that don't routinely update their apt cache
(probably including ~all Vagrant environments).
2020-03-23 13:47:17 -07:00
arpit551 dc6c362e4a api_docs: Replaced cgi.escape with html.escape.
cgi.escape is deprecated in python3.2 and removed in python3.8.
This function was unsafe because quote is false by default, hence
removed and replaced with a safer html.escape.
2020-03-23 10:28:21 -07:00
arpit551 258de497a8 setup_venv: Used get_venv_dependencies() to return correct VENV dependencies.
Used get_venv_dependencies function to return the correct dependencies
for RHEL, Centos, Fedora rather than importing them as separate
COMMON_YUM_DEPENDENCIES in provision and create-production-venv.
2020-03-23 10:28:21 -07:00
Dinesh dacca3a593 tests: Add `test-js-with-puppeteer` similar to `test-js-with-casper`.
Note that there are no puppeteer tests yet; this is just preparatory
infrastructure.
2020-03-22 11:42:14 -07:00
Dinesh 9e65f3f354 provision: Add provision support for puppeteer. 2020-03-22 11:42:14 -07:00
Steve Howell eb9a252ec9 populate_db, tests: Restrict emails in zulip realm.
We now restrict emails on the zulip realm, and now
`email` and `delivery_email` will be different for
users.

This change should make it more likely to catch
errors where we leak delivery emails or use the
wrong field for lookups.
2020-03-19 16:21:31 -07:00
Rohitt Vashishtha b25e02aa14 tests: Treat github.com/zulip links as external.
Tests for these links often result in rate-limiting from GitHub,
leading to the builds failing in Circle CI. We temporarily mark
github.com/zulip links as external to keep the builds passing.
2020-03-19 09:27:40 -04:00
Vishnu KS a48845c8fb tests: Create new test mode for generating stripe fixtures. 2020-03-17 16:46:56 -07:00
arpit551 e916d0b733 provision: Added provision support for Ubuntu 20.04(Focal).
Groonga does not have a ppa package for Focal yet so pgroonga is
built from the source.
Postgres 12 is used when os_version is 20.04.
2020-03-17 14:57:14 -07:00
arpit551 b0cb493850 setup_venv: Added get_venv_dependencies function.
Added a get_venv_dependencies() function in setup_venv.py which
returns VENV_DEPENDENCIES according to the vendor and os_version.
The reason for adding this function was because python-dev will be
depreciated in Focal but can be used as python2-dev so when adding
support for Focal VENV_DEPENDENCIES should to be os_version dependent.
2020-03-17 14:57:14 -07:00
arpit551 3963b6740a provision: Corrected how scripts/lib/build-* is included for sha1 sum check.
There were two problems with the previous code-
1) The code glob.glob("scripts/lib/build-") should be
   glob.glob("scripts/lib/build-*) otherwise it would always return [].
2) The part of the code where we included scripts/lib/build-* for sha1 sum
   check would only run when debian is not in os_families(). This wasn't
   correct as we could have a situation where we have to build pgroonga
   from source even in case of debian and so we need to improve the
   condition on it.
Now since we only have build-pgroonga there its better to just directly hash
its content with the condition of BUILD_PGROONGA_FROM_SOURCE.
2020-03-17 14:57:14 -07:00
Mateusz Mandera d00a579318 Revert "test-run-dev: Use a pipe rather than polling a log file."
This should fix spurious failures, where test-run-dev would occasionally
freeze. What exactly about these changes was causing that is still to
be potentially investigated. This is merely meant as a fix to the
failures.
This reverts commit 19429c3ad7.
2020-03-17 14:16:18 -07:00
Vishnu KS 1c6435d4cc validator: Optionally record a type_structure attribute.
We plan to use these records to check and record the schema of Zulip's
events for the purposes of API documentation.

Based on an original messier commit by tabbott.

In theory, a nicer version of this would be able to work directly off
the mypy type system, but this will be good enough for our use case.
2020-03-06 17:07:14 -08:00
Steve Howell e67be55152 node tests: Actually test `translations.js`.
Before this test, we were validating the behavior
of `i18next`, but we weren't validating our light
layer that sits on top of `i18next`, which currently
resides in the slightly misnamed `translations.js`
file.

The translations module is now so small that I'll
just quote it verbatim here:

    import i18next from 'i18next';

    i18next.init({
        lng: 'lang',
        resources: {
            lang: {
                translation: page_params.translation_data,
            },
        },
        nsSeparator: false,
        keySeparator: false,
        interpolation: {
            prefix: "__",
            suffix: "__",
        },
        returnEmptyString: false,  // Empty string is not a valid translation.
    });

    window.i18n = i18next;

We now just do `zrequire('translations')` to initialize
the `i18next` library, which allows us to have simpler
test setup and to actually exercise the above call to
`i18next.init`.

This change now gives us 100% line coverage of `translations.js`,
which of course isn't that hard to acheive (see above).
2020-02-29 12:19:51 -08:00
Anders Kaseorg 687553a661 setup_path_on_import: Replace with setup_path function.
isort 5 knows not to reorder imports across function calls, so this
will stop isort from breaking our code.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 15:40:21 -08:00
Anders Kaseorg 1cdab5ae61 emoji: Resolve emoji sprite sheets and stylesheets through Webpack.
This gives them cache-compatible URLs, and also avoids some extra
copies of the sprite sheet images.

Comments on the Octopus emoji added by tabbott.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 14:43:46 -08:00
Anders Kaseorg 197084ab93 generate_emoji_names_table: Update SORTED_CATEGORIES.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 14:30:31 -08:00
Anders Kaseorg b474fa0e37 webpack: Generalize debug-require-webpack-plugin interface.
Now the caller simply imports the debug ‘require’ function as a
module, deciding for itself how to expose it and with what name (in
our case, we expose it as ‘require’ with expose-loader).  Also, remove
a stray console.log.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-25 14:01:06 -08:00
Anders Kaseorg 46e14d1c40 webpack: Expose a version of require() for use in the browser console.
This adds a global require() function that makes JS modules accessible
to the browser console without adding them to the global window
object:

» const typeahead = require("./static/shared/js/typeahead");
» typeahead.popular_emojis
Array(6) [ "1f44d", "1f389", "1f642", "2764", "1f6e0", "1f419" ]

The list of known modules is exposed via the keys of require.ids
object.

This will allow us to migrate more modules to ES6 without losing
access to this debugging functionality.

I’ll probably upload this plugin to NPM at some point, but I figured
I’ll let it bake in-tree first.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-24 23:18:05 -08:00
Steve Howell 5e8279c2fb refactor: Extract settings_config.
This moves some code from settings_display.js
into the new module settings_config.js.

Extracting this module breaks some dependencies
on settings_display.js (which has some annoying
transitive dependencies, including jQuery).

In particular this isolates stream_data from
from settings_display.js.

Two of the three structures that we moved here
weren't even directly used by settings_display.js,
since we do a lot of rendering in the modules
admin.js and setting.js.

We make get_all_display_settings() a function
to avoid a require-time dependency on page_params.

Breaking the dependencies simplifies a few
node tests.

Most of the node test complexity came from the
following commit in March 2019:

5a130097bf

The commit itself seems harmless enough, but
dependencies can have a somewhat "viral" nature,
where making stream_data depend on settings_display
caused us to modify four different node tests.
2020-02-21 12:06:31 -08:00
Tim Abbott 8aae02de68 emoji: Add a block comment explaining spritesheet CSS. 2020-02-21 10:26:57 -08:00
Steve Howell af7923c557 refactor: Extract get_square_size() for emoji sprites.
This cleans up a few things:

    - just yield values so we don't have to do
      tedious max logic
    - use values() instead of items() for
      skin_variations loop

In the ideal world the emoji.json would reduce this
code to `get_square_size = lambda data: data['square_size']`,
but I don't think we can get the square size explicitly.
2020-02-21 10:21:12 -08:00
Steve Howell da1ce9a577 emoji sprites: Avoid hard coding CSS percentages.
This commit changes the calculation of the
background-size parameter that we use to
render emojis from sprite sheets.

In particular, it now makes the parameter
match the sizes of our latest sprite
sheets from Twitter/Google.

This should fix the geometry aspect of #13959,
but we also need to fix some issues with the
cache being sticky.

There is also some minor cleanup:

    - Remove obsolete -moz/-webkit CSS.
    - Remove needless precision in percentages.
    - Fix the transposed nrows/ncols names.
    - Add extensive commenting.

Finally, we add a minor bump to the provision
number.  This commit should be merged in the
same series as the other fix for this issue,
which will probably have a major bump, and we'll
need to rebase this appropriately.
2020-02-21 10:21:12 -08:00
Anders Kaseorg 9d598d95a6 puppet: Fix puppet-lint warning.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-20 19:51:48 -08:00
Tim Abbott aaee506fb2 provision: Always build pygments_data.json if missing.
While it's a bit of extra complexity to do this check, which I'm not
excited about, we've had multiple folks spend significant time being
confused rebasing past d7d8632525 into
deleting `pygments_data.json`, with provision not rebuilding it, so
this seems worth merging as a transitional fix even if we decide to
remove it in 2 months.
2020-02-20 10:25:11 -08:00
Mateusz Mandera 7293862eae tests: Keep temporary .coverage files in var/ directory. 2020-02-19 14:36:17 -08:00
Mateusz Mandera 782353f0e1 tests: Ensure coverage data files get cleaned up at the end.
Without calling cov.erase() the data file seems to persist and even
pollute future test runs if not removed. Registering an atexit handler
seems like a good, and reasonably clean way to ensure the cleanup
happens.
Fixes #13933.
2020-02-19 14:36:17 -08:00
Anders Kaseorg 7990ef2d52 register: Spell Greek Ακμή with Greek alpha, not Latin A
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-18 17:41:43 -08:00
Anders Kaseorg 4889a0486d tests: Compile Handlebars templates with source maps.
This allows us to collect coverage for Handlebars templates, and also
improves the readability of Handlebars-related stack traces.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-18 07:38:46 -05:00
Mateusz Mandera 36a8e61e67 run-dev: Run process_queue with DJANGO_AUTORELOAD_ENV.
In Django 2.2 the autoreload system has changed.
DJANGO_AUTORELOAD_ENV env variable should be set when calling code
that'll use the autoreloader. Otherwise there's some kind of race
condition in the autoreload code when SIGINT is sent, where
restart_with_reloader() (called only if the env variable isn't set)
has the subprocess module calling p.kill() on a process that's already
exited, raising ProcessLookupError and printing an ugly traceback. This
causes non-deterministic test-run-dev failures.
2020-02-17 13:06:50 -05:00
Vishnu KS 5bab2a3762 upload: Replace jQuery filedrop with Uppy. 2020-02-13 16:43:19 -08:00
Tim Abbott e2892a88c1 channel: Discard all HTTP responses while reloading.
We used to have a block of code doing this just in the presence
endpoint because that's where we'd had error-handling problems with it
not being present, but it seems more correct for it to run
unconditionally on all HTTP requests.

This requires adding a dependency of channel on reload_state, which we
record in the webpack configuration for now.
2020-02-13 15:45:39 -08:00
Tim Abbott 906160f1a3 presence: Re-introduce data filtering when offline.
This should return us to a situation where we won't get blueslip
browser error reporting for users created while a device was offline
just before it reloads.
2020-02-13 15:45:39 -08:00
Hashir Sarwar eb23c6fa6c test_fixtures: Clean up interface for `template_database_status()`.
1) Created a new class `DatabaseType` and access its objects inside
`template_database_status()` instead of sending five arguments with
default values.

2) Made `check_files` and `setting_name` local variables instead of
function parameters since they had same value(None) for every call.

Fixes #13845.
2020-02-12 11:07:10 -08:00
Anders Kaseorg d7d8632525 pygments_data: Replace JS module with JSON module.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-12 10:09:12 -08:00
Anders Kaseorg e257253e64 emoji_codes: Replace JS module with JSON module.
webpack optimizes JSON modules using JSON.parse("{…}"), which is
faster than the normal JavaScript parser.

Update the backend to use emoji_codes.json too instead of the three
separate JSON files.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-12 10:09:12 -08:00
Steve Howell 7ec5fbab2e node tests: Remove templates test.
I believe we can remove these and rely on
other parts of our testing/code-review
to ensure template quality.

These tests never really exercised our
app code, as evidenced by us not regressing
any of the 100%-line-coverage files.

We have a couple other ways that we verify
the correct format of the templates:

    - webpack (can they compile?)
    - check-templates (are they nicely indented?)

For deep testing, we have Casper, which
exercises most of our most important templates
in some meaningful way.

I think it's pretty rare that we get bugs
now that are directly caused by bad templates,
and an even smaller subset of them would
have been caught by the node tests.

If that trend changes in the future, I would prefer to
just do something "greenfield" to address
any common problems rather than resurrect
this code, but we could always resurrect it
from git.

The template node tests did check a little bit of
detail about which fields are there, but not
in an integrated way, so that aspect of the tests
wasn't very useful either.
2020-02-12 09:58:23 -08:00
Chris Heald 2ca447c1a5
provisioning: Support Ubuntu 19.10 eoan.
This adds Ubuntu 19.10 as a valid provisioning target.

The release test in setup-apt-repo was changed from a list of values to
a regex check for brevity.
2020-02-12 09:56:42 -08:00
Greg Price 1e2b558b01 reset-to-pull-request: Add a usage message.
Every CLI program should have a usage message.

Also add a mention in the `push-to-pull-request` usage message of
its participation in the `refs/remotes/pr/` pseudo-remote feature.
2020-02-11 14:45:17 -08:00
Greg Price e3d843baf9 push-to-pull-request: Give a more detailed error on URL parse failure.
This simplifies debugging when this error fires, e.g. for finding
an issue like the one we just used `git remote get-url` to fix.
2020-02-11 14:45:17 -08:00
Greg Price e7a82e732e push-to-pull-request: Use `git remote get-url`.
This gives us the right behavior when using the `url.*.insteadOf`
mechanism for aliases in Git remote URLs.  For example, if
one's ~/.gitconfig has:

  [url "git@github.com:"]
      insteadOf = gh:

then `git remote add upstream gh:zulip/zulip` will work great, as
the nice, short, mnemonic `gh:` prefix gets expanded to the more
finicky `git@github.com:`.  I use just such a prefix routinely.

But the feature does require that scripts go through the right
abstractions.  In particular `git remote get-url`, since Git 2.7
(from 2016), exists for exactly this reason.  A plain `git config`
command bypasses the expansion, getting the verbatim `gh:...`
version, which doesn't work.

So, switch to that.

As a bonus, we get to behave correctly if for some reason the user
has configured a push URL distinct from the fetch URL for this
remote, just by adding `--push`.  With `git config`, we'd have had
to manually implement the fallback from `remote.upstream.pushUrl` to
`remote.upstream.url` in order to properly handle that case.
2020-02-11 14:45:17 -08:00
Steve Howell 6922eef380 signups: Use internal_send_stream_message().
We prefer this to internal_send_message().

We are trying to deprecate `internal_send_message`,
which has extra moving parts related to
`extract_recipients` and `Addressee.legacy_build`.

There are two chunks of code that I touch here
that look pretty similar, but I'm not quite
sure they're worth de-duplicating, since they
use different topics and different message
content.
2020-02-10 15:45:13 -08:00
Steve Howell b33552997e cross realm bots: Simplify notify_new_user.
Instead of having `notify_new_user` delegate
all the heavy lifting to `send_signup_message`,
we just rename `send_signup_message` to be
`notify_new_user` and remove the one-line
wrapper.

We remove a lot of obsolete complexity:

    - `internal` was no longer ever set to True
      by real code, so we kill it off as well
      as well as killing off the internal_blurb code
      and the now-obsolete test

    - the `sender` parameter was actually an
      email, not a UserProfile, but I think
      that got past mypy due to the caller
      passing in something from settings.py

    - we were only passing in NOTIFICATION_BOT
      for the sender, so we just hard code
      that now

    - we eliminate the verbose
      `admin_realm_signup_notifications_stream`
      parameter and just hard code it to
      "signups"

    - we weren't using the optional realm
      parameter

There's also a long ugly comment in
`get_recipient_info` related to this code
that I amended for now.
We should try to take action in a subsequent
commit.
2020-02-10 15:45:13 -08:00
Anders Kaseorg 719546641f js: Convert a.indexOf(…) !== -1 to a.includes(…).
Babel polyfills this for us for Internet Explorer.

import * as babelParser from "recast/parsers/babel";
import * as recast from "recast";
import * as tsParser from "recast/parsers/typescript";
import { builders as b, namedTypes as n } from "ast-types";
import K from "ast-types/gen/kinds";
import fs from "fs";
import path from "path";
import process from "process";

const checkExpression = (node: n.Node): node is K.ExpressionKind =>
  n.Expression.check(node);

for (const file of process.argv.slice(2)) {
  console.log("Parsing", file);
  const ast = recast.parse(fs.readFileSync(file, { encoding: "utf8" }), {
    parser: path.extname(file) === ".ts" ? tsParser : babelParser,
  });
  let changed = false;

  recast.visit(ast, {
    visitBinaryExpression(path) {
      const { operator, left, right } = path.node;
      if (
        n.CallExpression.check(left) &&
        n.MemberExpression.check(left.callee) &&
        !left.callee.computed &&
        n.Identifier.check(left.callee.property) &&
        left.callee.property.name === "indexOf" &&
        left.arguments.length === 1 &&
        checkExpression(left.arguments[0]) &&
        ((["===", "!==", "==", "!=", ">", "<="].includes(operator) &&
          n.UnaryExpression.check(right) &&
          right.operator == "-" &&
          n.Literal.check(right.argument) &&
          right.argument.value === 1) ||
          ([">=", "<"].includes(operator) &&
            n.Literal.check(right) &&
            right.value === 0))
      ) {
        const test = b.callExpression(
          b.memberExpression(left.callee.object, b.identifier("includes")),
          [left.arguments[0]]
        );
        path.replace(
          ["!==", "!=", ">", ">="].includes(operator)
            ? test
            : b.unaryExpression("!", test)
        );
        changed = true;
      }
      this.traverse(path);
    },
  });

  if (changed) {
    console.log("Writing", file);
    fs.writeFileSync(file, recast.print(ast).code, { encoding: "utf8" });
  }
}

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-10 14:08:12 -08:00
Steve Howell 3af8dc6c7d node tests: Add some color to output. 2020-02-09 22:07:51 -08:00
Steve Howell e1977f4680 node tests: Use EXEMPT_FILES for line coverage.
We now have 100% line coverage on 71 JS files.
This is thanks to about 150 people who have
contributed code to frontend/node_tests.

And then 126 files are still short of 100% line
coverage.

We now enforce line coverage with a set called
EXEMPT_FILES, which are the files for which
we do NOT expect to have 100% coverage.

Using an exemption list makes it so that adding
a new JS file to the project without 100% line
coverage will cause the build to fail.  This will
encourage folks to be intentional about their
lack of test coverage.
2020-02-09 22:07:51 -08:00
Steve Howell fe48ede9cf node tests: Fix bug with zero-coverage case.
If a file that had 100% coverage somehow regressed
to 0% coverage, we would report an error to the
console, but we weren't treating it as an actual
failure.

We've probably always had this bug, but it probably
rarely was an issue, since devs might have seen
the error locally, or hopefully whatever crazy
thing you did to totally remove coverage would
have had other symptoms.

If this was intentional, I suspect it might have
had something to do with wanting to get coverage
reports when you just run individual tests.  But
a while back we changed it so that when you run
individual tests, we don't do the line coverage
enforcement.
2020-02-09 22:07:51 -08:00
Vishnu KS 4572be8c27 api: Rename subject_links to topic_links.
Fixes #13588
2020-02-07 14:35:22 -08:00
Anders Kaseorg 02511bff1c js: Automatically convert _.each to for…of.
This commit was automatically generated by the following script,
followed by lint --fix and a few small manual lint-related cleanups.

import * as babelParser from "recast/parsers/babel";
import * as recast from "recast";
import * as tsParser from "recast/parsers/typescript";
import { builders as b, namedTypes as n } from "ast-types";
import { Context } from "ast-types/lib/path-visitor";
import K from "ast-types/gen/kinds";
import { NodePath } from "ast-types/lib/node-path";
import assert from "assert";
import fs from "fs";
import path from "path";
import process from "process";

const checkExpression = (node: n.Node): node is K.ExpressionKind =>
  n.Expression.check(node);
const checkStatement = (node: n.Node): node is K.StatementKind =>
  n.Statement.check(node);

for (const file of process.argv.slice(2)) {
  console.log("Parsing", file);
  const ast = recast.parse(fs.readFileSync(file, { encoding: "utf8" }), {
    parser: path.extname(file) === ".ts" ? tsParser : babelParser,
  });
  let changed = false;
  let inLoop = false;
  let replaceReturn = false;

  const visitLoop = (...args: string[]) =>
    function(this: Context, path: NodePath) {
      for (const arg of args) {
        this.visit(path.get(arg));
      }
      const old = { inLoop };
      inLoop = true;
      this.visit(path.get("body"));
      inLoop = old.inLoop;
      return false;
    };

  recast.visit(ast, {
    visitDoWhileStatement: visitLoop("test"),

    visitExpressionStatement(path) {
      const { expression, comments } = path.node;
      let valueOnly;
      if (
        n.CallExpression.check(expression) &&
        n.MemberExpression.check(expression.callee) &&
        !expression.callee.computed &&
        n.Identifier.check(expression.callee.object) &&
        expression.callee.object.name === "_" &&
        n.Identifier.check(expression.callee.property) &&
        ["each", "forEach"].includes(expression.callee.property.name) &&
        [2, 3].includes(expression.arguments.length) &&
        checkExpression(expression.arguments[0]) &&
        (n.FunctionExpression.check(expression.arguments[1]) ||
          n.ArrowFunctionExpression.check(expression.arguments[1])) &&
        [1, 2].includes(expression.arguments[1].params.length) &&
        n.Identifier.check(expression.arguments[1].params[0]) &&
        ((valueOnly = expression.arguments[1].params[1] === undefined) ||
          n.Identifier.check(expression.arguments[1].params[1])) &&
        (expression.arguments[2] === undefined ||
          n.ThisExpression.check(expression.arguments[2]))
      ) {
        const old = { inLoop, replaceReturn };
        inLoop = false;
        replaceReturn = true;
        this.visit(
          path
            .get("expression")
            .get("arguments")
            .get(1)
            .get("body")
        );
        inLoop = old.inLoop;
        replaceReturn = old.replaceReturn;

        const [right, { body, params }] = expression.arguments;
        const loop = b.forOfStatement(
          b.variableDeclaration("let", [
            b.variableDeclarator(
              valueOnly ? params[0] : b.arrayPattern([params[1], params[0]])
            ),
          ]),
          valueOnly
            ? right
            : b.callExpression(
                b.memberExpression(right, b.identifier("entries")),
                []
              ),
          checkStatement(body) ? body : b.expressionStatement(body)
        );
        loop.comments = comments;
        path.replace(loop);
        changed = true;
      }
      this.traverse(path);
    },

    visitForStatement: visitLoop("init", "test", "update"),

    visitForInStatement: visitLoop("left", "right"),

    visitForOfStatement: visitLoop("left", "right"),

    visitFunction(path) {
      this.visit(path.get("params"));
      const old = { replaceReturn };
      replaceReturn = false;
      this.visit(path.get("body"));
      replaceReturn = old.replaceReturn;
      return false;
    },

    visitReturnStatement(path) {
      if (replaceReturn) {
        assert(!inLoop); // could use labeled continue if this ever fires
        const { argument, comments } = path.node;
        if (argument === null) {
          const s = b.continueStatement();
          s.comments = comments;
          path.replace(s);
        } else {
          const s = b.expressionStatement(argument);
          s.comments = comments;
          path.replace(s, b.continueStatement());
        }
        return false;
      }
      this.traverse(path);
    },

    visitWhileStatement: visitLoop("test"),
  });

  if (changed) {
    console.log("Writing", file);
    fs.writeFileSync(file, recast.print(ast).code, { encoding: "utf8" });
  }
}

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-07 14:09:47 -08:00
Anders Kaseorg 811e128787 check-openapi: Fix lint errors and remove lint exclusion.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-07 14:09:47 -08:00
Anders Kaseorg b3f63262af lint: Delegate console.log check to ESLint.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-07 14:09:47 -08:00
Anders Kaseorg 61bf698185 requirements: Fork pip.txt to pip2.txt for Python 2.
The current version of setuptools no longer supports Python 2.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg d2e07ea51b setup_venv: Replace virtualenv_args list with python2 bool.
Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-05 12:38:10 -08:00
Anders Kaseorg fa558de6d8 commit-message-lint: Detect the upstream remote name.
My upstream remote is named origin, so commit-message-lint was always
complaining at me.  Detect the right remote name from the output of
`git remote -v`.

Signed-off-by: Anders Kaseorg <anders@zulipchat.com>
2020-02-04 13:57:51 -08:00
Mateusz Mandera 4c5a8e6f0c queue: Remove missedmessage_email_senders. 2020-01-31 12:13:51 -08:00
Mateusz Mandera a583da67b3 test-queue-worker-reload: Fix expected number of workers. 2020-01-31 12:13:51 -08:00
Steve Howell eeee6edf41 pm_list: Simplify redraws for Private Messages.
We now use vdom-ish techniques to track the
list items for the pm list.  When we go to update
the list, we only re-render nodes whose data
has changed, with two exceptions:

    - Obviously, the first time we do a full render.
    - If the keys for the items have changed (i.e.
      a new node has come in or the order has changed),
      we just re-render the whole list.

If the keys are the same since the last re-render, we
only re-render individual items if their data has
changed.

Most of the new code is in these two modules:

    - pm_list_dom.js
    - vdom.js

We remove all of the code in pm_list.js that is
related to updating DOM with unread counts.

For presence updates, we are now *never*
re-rendering the whole list, since presence
updates only change individual line items and
don't affect the keys.  Instead, we just update
any changed elements in place.

The main thing that makes this all work is the
`update` method in `vdom`, which is totally generic
and essentially does a few simple jobs:

    - detect if keys are different
    - just render the whole ul as needed
    - for items that change, do the appropriate
      jQuery to update the item in place

Note that this code seems to play nice with simplebar.

Also, this code continues to use templates to render
the individual list items.

FWIW this code isn't radically different than list_render,
but it's got some key differences:

    - There are fewer bells and whistles in this code.
      Some of the stuff that list_render does is overkill
      for the PM list.

    - This code detects data changes.

Note that the vdom scheme is agnostic about templates;
it simply requires the child nodes to provide a render
method.  (This is similar to list_render, which is also
technically agnostic about rendering, but which also
does use templates in most cases.)

These fixes are somewhat related to #13605, but we
haven't gotten a solid repro on that issue, and
the scrolling issues there may be orthogonal to the
redraws.  But having fewer moving parts here should
help, and we won't get the rug pulled out from under
us on every presence update.

There are two possible extensions to this that are
somewhat overlapping in nature, but can be done
one a time.

    * We can do a deeper vdom approach here that
      gets us away from templates, and just have
      nodes write to an AST.  I have this on another
      branch, but it might be overkill.

    * We can avoid some redraws by detecting where
      keys are moving up and down.  I'm not completely
      sure we need it for the PM list.

If this gets merged, we may want to try similar
things for the stream list, which also does a fairly
complicated mixture of big-hammer re-renders and
surgical updates-in-place (with custom code).

BTW we have 100% line coverage for vdom.js.
2020-01-30 13:11:32 -08:00
Mateusz Mandera a25f00a69c populate_db: Extract some functions to server_initialization.py. 2020-01-29 17:26:45 -08:00
Tim Abbott 7bf3312114 api: Document new get_messages oldest/newest API feature.
While we're at it, we make the examples more sensible.
2020-01-29 12:14:06 -08:00
Steve Howell b945cfd291 test-js-with-node: Extract run_tests_via_node_js(). 2020-01-27 16:32:11 -08:00
Steve Howell cc07bf04b7 test-js-with-node: Move constants toward top. 2020-01-27 16:32:11 -08:00