Instead, build them automatically when provision the development
environment and in update-prod-static.
(imported from commit aac8dfeaafbe872c113e5f2b6bd8f655a1af36f2)
It doesn't have any sensitive data since that lives in a separate
configuration file, and it's potentially useful.
(imported from commit 094e315439f8bd23ad07a8c2bc7d9776c8c7f096)
The tarball build process runs in DEVELOPMENT mode, assuming it is run
on a dev VM (since then there is no /etc/zulip directory). Commit
d067bcfe9d71 made settings.py import local_settings_template.py in
DEVELOPMENT mode (then "not DEPLOYED"), not local_settings.py.
(imported from commit 9a08138d748dfca9c4ab8b366bee5c2fb96c25af)
Just importing zerver.lib.cache creates a file memcached_prefix that
is mode 0444, so we need to use -f or rm will prompt about whether to
remove it. Not sure why this is apparently a new issue.
(imported from commit 93c5140b66992339859e2b204c200d1dd7a35f2d)
This commit loses some indexes, unique constraints etc. that were
manually added by the old migrations. I plan to add them to a new
migration in a subsequent commit.
(imported from commit 4bcbf06080a7ad94788ac368385eac34b54623ce)
We don't need to check whether the user exists before creating it:
CREATE USER failing is fine.
(imported from commit e8b2bc5495e328ee30d15445a566c0edff2f069d)
If we run provision.py a second time, there will already be
zulip/zulip_test users, so the CREATE USER will fail and the password
won't get updated to the newly generated value. By creating the user
and setting the password in two commands, we allow the creation to
fail without affecting whether the password is set.
Also the quoting for updating .pgpass was wrong.
(imported from commit 5e249813c17cb4829e4e4958e92aaa30563c5f96)
Sometimes I get the error "Selected message id not in MessageList"
when running the casper tests. I think it's probably when the test
user's home view does not contain any messages.
Ideally we would fix this in a way that guarantees that we generate
whatever messages the test suite needs...
(imported from commit 51a02da612dda88d60681b9e09cd6e6a2c39a470)
Source LOCAL_DATABASE_PASSWORD and INITIAL_PASSWORD_SALT from the secrets file.
Fix the creation of pgpass file.
Tim's note: This will definitely break the original purpose of the
tool but it should be pretty easy to add that back as an option.
(imported from commit 8ab31ea2b7cbc80a4ad2e843a2529313fad8f5cf)
The old language was confusing because "the interface" could refer to something
like eth0, but in actuality refers to the IP/hostname to listen on.
(imported from commit 4f77d72a4dfcdbe7e7747c6228975aa68dfbe6ac)
It's been very buggy for a while, has limited usefulness compared with
unread counts, and profiling over the weekend indicates that it's very
slow.
(imported from commit 716fe47f2bbec1bd8a6e4d265ded5c64efe2ad5c)
This doesn't change the alerting UI logic, it just turns
alert_words_ui into a module and calls the setup code from settings.js
when the settings page is rendered.
(imported from commit 05f95383b046086641280f82f648be58688efe61)
Before this change, the way we'd strip tags of punctuation
was just sort of messed up, because we'd strip the start tags
one way and strip the end tags another, and we had conditionals
for the different flavors of tags, instead of doing the stripping
when we already knew what flavor of tag we were dealing with.
(imported from commit 60c5ebd45e21b88bbfc98ff4b43dbbc6b32b38a1)
This makes it simpler to test between two VMs by allowing you to bind to
non localhost interfaces.
(imported from commit f70755533b52ff8c49fd916941d2210fb8c33b47)
Importing zerver.worker.queue_processors (which is needed to get the list of
workers to start) is slow because it, in turn, imports a bunch of stuff. So we
move the process of starting up queue processing workers into another script
that gets started in parallel with everything else.
(imported from commit 839bada6dc7b93825c69b0d8fd9fbe2de75eabee)
Add a helper to patch_global to change a global and then reset it to the
original value after a test file is complete.
(imported from commit 1b65ff6ea8693ad61b7f18f35dafa942429252a8)
In the early days of the node tests we didn't have an index.js
driver, so each test set it up its own environment. That ship
has long since sailed, so now we just require assert in index.js
as a global and make it so that the linter doesn't complain.
(imported from commit 1ded3d330ff40603cf4dd7c5578f6a47088d7cc8)
I apparently screwed up when backing up the process_loaded_for_unread
move in a way that just lost the function.
(imported from commit 91dfcf1abc85d439274cb8b0be380e9230942ebb)
The production database has a public schema. I thought we had dropped it, but
apparently not. We should match what exists in prod either way.
(imported from commit 1bf956360029ebbd59afc3cc30fca9a859343adf)
We do this by creating a new zulip{_test}_base database that only has the zulip
schema and the tsearch_extras extension. We then use that as a template when
creating zulip{_test}.
(imported from commit 8adb4b98410e4042a0187902e89c99561eac8c8f)
These classes are in test_hooks.py now. They still run as part of
the regular suite, so this is just to make it easier to navigate the
files.
JiraHookTests
BeanstalkHookTests
GithubV1HookTests
GithubV2HookTests
PivotalV3HookTests
PivotalV5HookTests
NewRelicHookTests
StashHookTests
FreshdeskHookTests
ZenDeskHookTests
(imported from commit 26a9572dd5170f9516e739d587a119bd1f87959a)
We changed our endpoint from "get_public_streams" in August, but the
API call whitelist was not updated.
(imported from commit 293c1da8e43c24ad8188ed2096a47992ad3a2c89)
Have run-dev.py watch for template changes by calling
`./tools/compile-handlebars-templates forever`. This doesn't
have much effect until the subsequent commit, but it does
alert users to broken templates.
(imported from commit 3fa5f403cabe0057f6f43180f1d09db669d98682)
The "forever" option causes the tool to continue looking for
template changes and, when they happen, to recompile them.
(imported from commit 2fa719a205f02c7c90cc071f99252148a888654f)
Before this change, we were compiling handlebars templates but then
still sometimes used the copy of compiled.js from the previous deploy,
which made zero sense. Since compiling is super fast, we continue to
compile during every deployment, and now we actually use the results
of compiling. Forcing a recompile every time avoids pitfalls like
failing to notice deleted files or failing to notice a Handlebars
upgrade.
(imported from commit 675932428ec420bfe0fd5a5c748a85600206764c)
postgres-init-db was expecting to be located two directories below the
project root. It is only one directory below the root of the project so
remove a set of .. .
(imported from commit 228b47ac2539caf2b6cd12a1b5f399534cf0c866)
This replaces the --noworkers option; the new --minimal option
starts a couple "essential" workers.
(imported from commit 4ca08709052c47257bc0448e51760edb4969d92e)
This allows us to avoid a circular import when importing models.py
from inside bugdown for the realm-filters-from-database branch.
(imported from commit 7de85b54243132ade6818b080abdc8c5e8ad84f5)
There are now 2 cases for narrowing:
1. We narrowed, but only backwards in time (ie no unread were
read). In this case, try to go back to exactly where we were before
narrowing. This behavior is unchanged.
2. We read some unread messages in a narrow. Instead of going back to
where we were before the narrow, go to our first unread message (or
the bottom of the feed, if there are no unread messages). This is new.
This means that after catching up through the sidebar, on returning
home you'll be at the bottom of your feed.
Searching for the first unread message in a message list with 40,000
messages only takes 17ms according to:
function timeit() {
var t0 = new Date().getTime();
_.find(current_msg_list.all(), unread.message_unread);
var t1 = new Date().getTime();
console.log('Find first unread: ' + (t1 - t0) + ' ms');
}
(imported from commit 87c467578a2cced0aa976d8ae2924371b85d2445)
This basically prints out a template JSON data structure to
be used with a handlebar template that you specify on the command
line. (You can actually supply multiple files, too.)
Example usage:
$ ./tools/get-handlebar-vars static/templates/tab_bar.handlebars
=== static/templates/tab_bar.handlebars
{
"tabs": [
{
"hash": "",
"title": "",
"active": "",
"icon": true,
"data": "",
"cls": ""
}
]
}
(imported from commit d7239fcae7d94038fa0e4b34c8b1208a1070ecbb)
This uses git ls-files -m, which will show modified and added files,
but it doesn't seem to show staged files, so buyer beware.
(imported from commit 6ecc1d5ee628deae17197addf5586f1f6bcd4b9c)
If you use persistent ssh connections, ssh'ing as admin will cause a sshd
process to hang around on the server, preventing us from deleting the admin
account. Therefore, we disable persistent connections for that ssh connection.
(imported from commit 2d043768417d20ef2f12695475a20b74bf3374de)
This requires a puppet apply on each of staging and prod0 to update
the nginx configuration to support the new URL when it is deployed.
(imported from commit a35a71a563fd1daca0d3ea4ec6874c5719a8564f)
When you upload a 2nd avatar to Zulip, the URL doesn't actually
change, so even new messages can show the old avatar, if your
browser is caching. We work against the cache by having the
"stamp" argument, which we vary at reload time and also when
we upload the new avatar. The browser still benefits from
cached images as new messages come in.
(imported from commit 84869c8d7f251c9f2498026a5e9e3b2451784879)
The check-handlebars-templates script now looks at most of our
back end templates to try and find imbalanced tags. This commit
fixes a bunch of the existing templates.
(imported from commit fad4a5d85d68160370dd588b41d6f125f64d198f)
Because git < 1.8.1 ignores lines with trailing slashes and
1.8.1.1 - 1.8.1.6 ignore lines without!
(imported from commit 8139a742f4a52ccb1bce4e06fb24c9626fdb01f2)
update-prod-static needs DEBUG=False. This also replaces our
local_settings.py before generating anything included in the tarball.
(imported from commit 890cd9d1a44acfd2c20e1662e0c68132c633d1b3)
We still need it in integrations, because those don't require Python
2.7, but we don't need it in any of our code that runs on internal
servers.
(imported from commit 3c340567f1a372dcb4206c6af9a6e5e18005b1b8)
The corporate "app" is not a full-fledged Django app, but it has
a urls.py and a templates directory. This commit creates the app
and moves the jobs pages into it. Localserver deployments will
not see any of the corporate code.
(imported from commit 35889c3cf92329258c30741fdfa564769a4fac1a)
The main use-case here is for ensuring that we don't deploy to master
while doing demos.
(imported from commit d3c8ba502052b352291548200032f39e0743b774)
Run the following commands as root before deploying this branch:
# /root/zulip/tools/migrate-server-config
# rm /etc/zulip/machinetype /etc/zulip/server /etc/zulip/local /etc/humbug-machinetype /etc/humbug-server /etc/humbug-local
(imported from commit aa7dcc50d2f4792ce33834f14761e76512fca252)
This moves the list of removed files from .gitattributes to
tools/build-local-server-tarball because static/ and tools/ are
necessary for update-prod-static, and it seemed best to keep the
entire list in one place.
(imported from commit 2a447cbde29e90d776da43bb333650a40d4d363c)
update-deployment has been replaced by upgrade-zulip for local server
instances, since it won't be running off a git repository, and
update-prod-static won't be needed since we plan on shipping minified
javascript.
When we deploy this, the deployment will fail, and then we'll need to
update the git checkout from which post-receive runs on git.zulip.net.
(imported from commit 86aaedbab09c60ae86ac1d0ae492d0d1bc45569f)
I switched narrow.by_subject and narrow.by_recipient to use the all_msg_list
instead of current_msg_list, since we wanted to be able to narrow to messages
specifically not in the current_msg_list. However, in searches which revealed
old messages outside the range of all_msg_list (which only has a single contiguous range),
this broke narrowing.
Let's use msg_metadata_cache instead.
(imported from commit 427f717484b4ae83d9bb4cc6e51ce17177d037fe)
Looking at the historical data, fewer than 50% of active users have
completed the checklist, which means that it is just persistent
clutter. We also have other better ways of encouraging people to send
traffic and get the apps now.
This commit removes both the frontend UI and backend work but leaves
the db row for now for the historical data.
(imported from commit e8f5780be37bbc75f794fb118e4dd41d8811f2bf)
We need to maintain this by labeling the files that we don't want to
ship with our local server tarballs in the .gitattributes file.
(imported from commit e29f38c477a4cdfd80fbb8e4e95c611b34f3b212)
Check that settings.html has at least balanced tags, and
automate the checking of those tags.
(imported from commit 35e9be269caa211803d64f2b54cb0287e13707b3)
We need to do a puppet apply immediately after deploying this, or our
rabbitmq consumer state files will become stale.
(imported from commit 18696a5a8b1ff431425d1f71c208acc9bf0694f2)
This requires no changes in production, but is tagged as manual to
remind developers that they need to edit and run the tools/migrate-db
script to fix up their local database instances.
(imported from commit fbf764fb61592ef994d6d2ad56edad65ff01f14b)
This commit must be simultaneously deployed on both staging and
prod0. It also requires completely taking down the app.
To deploy these changes, do:
* check out this commit at /root/zulip on postgres0, postgres1, staging, and prod0
* stop the process_fts_updates job on postgres0 and postgres1
* stop the app on staging and prod0
* do a puppet apply on postgres0, postgres1, staging, and prod0
* move the new client certificates into place on staging and app
* move the new server certificates into place on postgres0 and postgres1
* reload the database config on postgres0 and postgres1 (this might
actually require a restart)
* run tools/migrate-db on postgres0 as root
* do a deploy through this commit on staging and prod0
* start the process_fts_updates job on postgres0 and postgres1
* do a puppet apply on nagios
(imported from commit 819bdd14326c1425e2d3041a491a8ca3b9716506)
New dependency: sockjs-tornado
One known limitation is that we don't clean up sessions for
non-websockets transports. This is a bug in Tornado so I'm going to
look at upgrading us to the latest version:
https://github.com/mrjoes/sockjs-tornado/issues/47
(imported from commit 31cdb7596dd5ee094ab006c31757db17dca8899b)
This may require just doing an mv on the home directory, plus changing
the home directory in /etc/passwd. It should of course be done carefully.
(imported from commit 660997d897ee6d33563af74f0fc5d4267a911755)
This requires doing an `mv` and then `puppet apply` on each of staging
and prod as part of the deployment process.
(imported from commit 5d0be64a3846f7151d2036d2e0b31049bc1c2dd2)
You can immediately apply the changes by passing '-f' or '--force' as
the first argument.
(imported from commit 91f00f6afd3c42e2b11f60e92fc20962bc952f0d)
This temporarily breaks the rabbitmq consumer checks for the
user_activity and notify_tornado queues because their state files
were renamed to match their queue names. It will be fixed for
staging in the next commit.
(imported from commit a6aaa330a1134d8ddffe8f4959deb12b219f241a)
The minify logic doesn't have an easy way to detect that you
removed a file since the last deployment.
(imported from commit 50d05fcdad382a586073c06d29d279433d1bba81)
people_list and people_dict include the feedback bot and anyone you've
cross-realm PM'd with. Useful for autocomplete, but not for admin and
stream settings views.
Fixes the UI part of Trac #1772.
(imported from commit cdefd4e86980447aad5190e7fc8ae3666d66e3c3)
This shows up when you're not running a Zephyr mirroring bot and lets
you use Webathena to have us run it. Obviously needs more docs.
Current problems include:
* supervisorctl reload ends up recreating /var/run/supervisor.sock
with the wrong permissions, so it only works once in a row before
you need to chmod that.
* /etc/supervisor/conf.d needs to be humbug-writeable; this is a clear
local root vulnerability
* This uses SSH and thus is kinda slow.
(imported from commit 7029979615ffd50b10f126ce2cf9a85a5eefd7a2)
This change will allow us to test the muting feature on
staging. Any topic named "muted" will automatically be
muted. You can also mute any other topic on the console:
muting.mute_topic('devel', 'ios');
current_msg_list.rerender();
More UI around this experiment will be coming soon, as well
as support for muting entire streams.
The muting module keeps track of which topics are muted, but a
user can expand muted messages, and once that happens, the
messages are marked with the "force_expand" flag that gets
persisted to the back end.
Muted messages are rendered in similar fashion to the summarized
rows, and as part of unifying some of that code, we have
made it so that expanding a summarized section doesn't remove
individual flags related to summaries; instead, the messages
get the force_expand flag set.
(imported from commit acee4190e63813d46850415c41ff8ebfae4a6953)
Have ui.set_presence_list() only touch the presence list.
Before this change, it was calling update_unread_counts(), which
has a bunch of side effects unrelated to the presence list.
(imported from commit 690f754d78874a03fa36f8ff8765d5a63e431d28)
This was broken in two ways:
1. Commit ad59d6f78042ce89, "Make the left sidebar and right sidebar
more consistent", pushed last Monday, changed the markup for the right
sidebar without changing a selector in stream_list.js that was looking
for the old markup.
2. Even then, whenever new user presence information came in, we would
rerender the user list and blow away the unread counts. This commit
patches around that by updating unread counts after rendering the user
list. I'm not sure what broke this or how it was working before.
(imported from commit 53ed40139e257e44411e918d1ecdce3a49e9ee51)
The functions add_dependencies() and set_global() are convenience
methods that allow you to modify the global namespace while
the current file is running but then have it be cleaned up
by index.js when you're done.
(imported from commit f75b8a10c19f773a8d2d3a8fa4bc39b1679566fe)
To use, make sure your Node installation is up to date and install
istanbul with npm:
sudo npm install -g istanbul # 'sudo' may be optional on OSX
Then run
tools/test-js-with-node cover
and navigate to coverage/lcov-report/js/index.html in a browser.
(imported from commit 2da4894d1725e2f9540b3895304246e3cd138f6c)
There are no functional changes; you can still use the shell script
tools/test-js-with-node. It just delegates now to the new index.js to
iterate through all the other .js files in the test directory and run
them. This sets the stage for Istanbul to correctly compute test
coverage.
(imported from commit 6f521c78b7a314d010fa113f9c2c971ab999b637)
Users were getting confused about why the unread count in the sidebar
/ notification bar / Dock was different from what the bankruptcy modal
said, so only show them the true server count until they've made a
decision.
(imported from commit 71d376cd4a85749ccf49936b251e6b8ac21361b7)
Move zulip.subject_dict into composebox_typeahead.seen_topics,
and encapsulate the use of seen_topics inside composebox_typeahead
with add_topic() and topics_seen_for().
(imported from commit 2bc2d1714fabdc07a661cbf815d14b36a08990e2)
We now show a list of users and allow you to deactivate a user using the
same process as `python manage.py deactivate_user`.
We add a new menu item accessible from the gear icon which will eventually
have much more than just this, but we have a good start here.
Here we also add a property to UserProfile which determines whether you're
eligible to access the administration panel, and then have code which shows
the menu option if so.
This introduces a new JS file, admin.js.
(imported from commit 52296fdedb46b4f32d541df43022ffccfb277297)
We instead implemented the ~desired functionality here using the
API and a bot to make a totally read-only, static, slowly-updating
view into the Zuliverse.
This is the moral equivalent of reverting deb035b4c702fcdb0e660ed549fe74c682abb6d9
(imported from commit 9d743fe82f197b37f005e5a038f77cc4b8566024)
1) The class Filter now lives in its own module.
2) The function canonicalized_operators() is now a class method on Filter.
3) The function message_in_home moved to filter.js and became private.
4) Various calling code had to change, of course.
5) Splitting out Filter helped simplify a few tests.
(imported from commit e41d792b46d3d6a30d3bd03db0419f129d0a2a7b)
The compose_fade has three public exports:
set_focused_recipient
unfade_messages
update_faded_messages
All code was pulled directly from compose.js, except for the
one-line setter of set_focused_recipient. The focused_recipients
variable that used to be in compose.js was moved to compose_fade.js,
hence the need for the setter.
(imported from commit 462ca5d0d0bd58612d0197f3734a8c78de8c6d30)
"Kiosk mode" is a "read-only" Zulip suitable for embedding into
an iframe on another site. I say "read-only" in quotation marks,
because the account is still a fully-fledged active account on
the server, and we just tear out a bunch of stuff in Javascript
(that a malicious user could easily re-enable).
So in that sense, it's not actually safe in security-sensitive
environments -- malicious users logged in via kiosk mode
can do anything the kiosk-mode user can do.
(We need this functionality for the customer3 realm specifically;
we'll possibly just tear this code back out once that experiment
has run its course.)
(imported from commit deb035b4c702fcdb0e660ed549fe74c682abb6d9)
util.enforce_arity takes a function and returns a new version which
throws an error if an incorrect number of arguments (as determined by
the function prototype) are passed.
(imported from commit 20e69a6dc7b6f8455726ab4fae8d5b7b04dc4103)
The test-all script now calls the symlink, and the run script
has been cleaned up to be symlink friendly.
(imported from commit 8abb5c1e5744416e94ff843e50c53e0d0f7e1316)
This is a pure refactoring that mostly just moves code from
subs.js to the new stream_color.js and updates module references
accordingly. In order to prevent introducing some exports,
update_stream_color was given an additional "sub" parameter
and update_stream_sidebar_swatch_color was given an "id"
parameter.
Killed off unused initial_color_fetch var.
(imported from commit b7644ce67f50d31fb46f564d758d661eea776aa6)
This includes a hack to preserve humbug/backends.py as a symlink, so
that we don't need to regenerate all our old sessions.
(imported from commit b7918988b31c71ec01bbdc270db7017d4069221d)
This needs to be deployed to both staging and prod at the same
off-peak time (and the schema migration run).
At the time it is deployed, we need to make a few changes directly in
the database:
(1) UPDATE django_content_type set app_label='zerver' where app_label='zephyr';
(2) UPDATE south_migrationhistory set app_name='zerver' where app_name='zephyr';
(imported from commit eb3fd719571740189514ef0b884738cb30df1320)
For now, we just get emails about referrals that we have to follow up
on manually.
I don't love using the name "referral" in some places and "invite" in
others, but we already use the verb "invite" to mean something else
and "invite" is a canonical noun.
(imported from commit 0814c18395952fcdef234c1584984f71ca1b6f37)
We always intended to move the pointer up when you were at the
top of the viewport and scrolling up, and we always intended to
move the pointer down when you were at the bottom of the viewport
and scrolling down. We didn't intend to move the pointer up
when you were at the bottom, or vice versa. This commit fixes
that bug, simplifies the code, and inlines the code into ui.js.
(imported from commit 77e2ace9d2fc1025e1349e3be13c76c3a397fd38)
This follows up on extracting code to js/search_suggestion.js, and all
the search tests apply to the new module.
(imported from commit dec6c8614c25b4f82da57edeaddc7cfef28260a5)
Moved 400+ lines out of search.js into search_suggestion.js. This
leaves search.js primarily responsible for lots of little DOM
interactions, whereas search_suggestion is more about data.
(imported from commit 53d08b29367c0172e483064f213538d45098279e)
I added our "static" directory to NODE_PATH for our JS unit tests.
This eliminates most of the verbosity in our require statements, but
it still requires us to explicitly call out "js" or "third"
subdirectories, which should make it a bit easier for folks reading
the tests to distinguish modules from js, third, or node itself.
(imported from commit b77a5283135d388d46f4b7e511acc59986f1a8ba)
In the future, this could come from the server to enable individual
experiments on a per-realm basis.
(imported from commit 8fd1ba1910b3cfd131f58bab8efbd11a42053bc3)
Previously, when the first `which` failed, the entire script would
fail with exist status 1 because we pass `-e` to the interpreter.
(imported from commit 601de3b3e3edd90110fc478f7874e644009d1b62)
This covers most of the module's functionality, with special
emphasis on lines that use underscore.js (_.each and _.filter).
(imported from commit 074181a0273286a258504be634bdd1cead2eecd5)
I haven't hooked this into test-all yet, but I did modify
message_tour.js to accomodate the test.
(imported from commit e58f595f4fe1160f539c18ec09dbe22eebf1f104)
When you return to Home, we normally restore your last
position when in Home, which might have been before the
first unread message. This is cumbersome for sidebar readers,
so now we keep track of messages that you visit while on a
tour away from Home, and when you return Home, we skip forward
any messages that were in the tour, landing on the last visited
message. This all happens before rendering.
(imported from commit 9124a231d94f153e283e5ea95e40c50a58406275)
When we push this to staging, we'll need to rename the bot in the
database and also pull on git.zulip.net.
(imported from commit 22b2397b197c8820f0e55daecd8f98d829e195bd)
Created up, down, page_up, page_down, to_home, and to_home pretty
mechanically from the old hotkey.code.
(imported from commit 5956b91c2e0122c6440f70db9b92f918c9b599aa)
* CC to code-review@zulip.com
* If domains are omitted, assume @zulip.com
* Don't set email or site explicitly, instead relying on ~/.humbugrc
(imported from commit 9700a0e3d6aac97c8030dcbaba7790018173929a)
We only want to change cases where we're talking about the hostname; HTTP
requests should still go to staging.humbughq.com for now.
Before this commit is deployed the hostname of staging.humbughq.com should
be changed to staging.zulip.net on the VM.
(the same for prod)
(imported from commit 7412530773f720ac227f40061c9ddb1a851e19bb)
This is a big change affecting lots of areas:
* Pipeline no longer deals with JS (though it still minifies CSS)
* A new script, tools/minify-js (called from update-prod-static),
minifies JavaScripts
* A command-line argument --prev-deploy, if passed to minify-js or
update-prod-static, is used to copy minified JS from a previous
deploy (i.e., a previous git checkout), if the source files have
not changed
* update-deployment passes --prev-deploy
* Scripts are now included with the minified_js template tag, rather
than Pipeline's compressed_js
Also, as a side benefit of this commit, our Handlebars templates will
no longer be copied into prod-static/ and accessible in production.
Unminification is probably broken, but, per Zev and Trac ticket #1377,
it wasn't working perfectly before this change either.
(Based on code review, this commit has been revised to:
* Warn if git returns an error in minify-js
* Add missing output redirects in update-prod-static
* Use DEPLOY_ROOT instead of manually constructing that directory
* Use old style formatting)
(imported from commit e67722ea252756db8519d5c0bd6a421d59374185)
The goal here is to make it easier to do ad hoc profiling on
our codebase, particularly by running tests.
(imported from commit 71da06feb3a369dec8dc4d8391f7f40e4c2d02ff)
Caveats:
- Since Chrome has trouble using W3C Notification when it's not
initiated by a user gesture, we try to use webkitNotification first.
- FF doesn't allow iconUrl to be of a different origin, so it won't display
our gravatars
(imported from commit c4f99ce6927a0d203d9f220d50b06737779bd7f8)
Per http://docs.python.org/2/library/os.html#os.getlogin, getlogin()
only works when you have an associated controlling tty.
This script didn't work previously because when we do deployments there
is no tty. Thus, we switch to the alternative mechanism for determining
the current username described on the page linked above.
(imported from commit 1dbcf98fd7248d20e501fd7fb22e1dbd306040fd)
If you're running this as a user other than "humbug" on a deployed server,
you're going to have a bad time.
Specifically, memcached won't work, and other undefined behaviour may
occur.
So here we add a check and error out if you run this script on an
app_frontend as non-"humbug".
(imported from commit a3d5f0f58ded42393c03f4d21b4650494fae418f)
Now that we have more than a month worth of data, it's nice to
actually just display it all.
(imported from commit f2ec027af81699d6937e52b18ac40973a54c349b)
Otherwise we end up running Tornado against the _previous_ version of
our code!
Testing of using supervisorctl to stop and then start a process shows
it takes about the same amount of time as doing a supervisorctl
restart, so there's no reason not to split the two commands apart and
make it super clear that nothing is running at the time that we move
the deployment symlink.
(imported from commit c38049da2bfc9fa94320a32dbf3240d1fcba67f7)
1) When you send a message, restore the focus to the composebox, targeted at the same recipient
2) If the composebox is completely empty and you press up or down, have that close the composebox and take the appropriate action
3) If you started the compose via a reply option (r, enter, click), don't refocus the composebox if the cursor has changed.
(imported from commit 84545e49d06959eb62e7fd2b22e1387383df6d1d)
Does a test-all using /tmp.
Not safe to run with other tests simultaneously.
You use it just like tools/review, and on a successful test-all
it will submit the code review, without tying up your git working directory.
It depends on new patches to the acrefoot-bulk_create_with_id-1.5.1 branch of
django. Updating this is the [manual] step required. Since test-all doesn't happen
on staging or prod, this is not required for this commit to be deployed, but other
django patches will help with the SSL connection errors we've been seeing.
(imported from commit 3fd3ff00240a2d648c4c54748a8a70616067ef7d)
Since update-deployment is run on the host being deployed to and only
has access to a recent clone of the git repository, it doesn't
necessarily have the old refs available for reverts.
(imported from commit 3652f58a7b165c805822bf6d8a4f0792c629e28e)
Previously our receive API bindings were broken in our API tarballs
because we weren't including the receive API bindings which they used.
This requires our deploying the built API tarball to the prod server
when we deploy it so that the link on /api isn't broken.
(imported from commit 14ecaab34556f4e29c72f4f567d8af73c89d6297)
With our currently rather slow deploys, this helps with the confusing
case where 2 deploys happen in quick succession and the second is
blocked by the lock.
(imported from commit 5a8bd449ecbe0825838faf72099b67f94db7797a)
Exit with a failing status if any of your
non-excluded JS/Python files have trailing whitespace.
(imported from commit 0aa184fa85a4a1ddeef6ea40dff57ce43c10e459)
This is a little hackish in that I'm using humbug-send rather than
importing humbug_git_config.py to get the API keys and all, but this
was quick to do and I think it's valuable for us to have this output
on our deploys.
(imported from commit 24d7954d04aadeff77ea9cdb76fdc8be055f3ac5)
Re-focuses on the compose box after a send, but only if
the compose-box was opened by responding to the message
at the cursor (by hitting "r", enter, or clicking on the message)
(imported from commit 8e7560c8ea31397b57b2bc3e2e7d9dd996226a6f)
Modified files need to be copied into the right place. The checkout
on git.humbughq.com also needs to be updated.
(imported from commit dbe9e05a0512e1f59c7819dd8d44c2c4e9c83bcf)
We let supervisor create the socket for us by making humbug-django a
fcig-program. Unfortunately, supevisor doesn't support putting
fcgi-programs in groups (see
https://github.com/Supervisor/supervisor/issues/148), so we have to
restart tornado and django separately.
To deploy, copy the config files over and restart nginx and
supervisor (via stopping and then starting it because restart is
broken). I believe the automated restart as part of
update-deployment will fail because of the way supervisor treats
programs in groups. If so, after restarting supervisor, you will
also need to run restart-server manually to fill the caches and then
delete the lock directory in humbug-deployments.
(imported from commit bfb5db7dd42dcbc4bfefa2944355b3cbb2ef9104)
The amount of process downtime during a supervisord-mediated restart
appears to be linear in the number of processes that are being
restarted. Therefore, restarting just Django and Tornado causes less
downtime than doing them at the same time as the other worker
processes.
(imported from commit 1fa9ef547bcd88caeec49800664e37d5f2fcb7a8)
Currently our test database is backed by sqlite; this commit moves
us to using postgres for our all database needs. This, in conjunction
with the patched django on github, allow us to have fewer hacks and
more true-to-life tests. It also sets the stage for testing the bulk_create
and schema search_path patches made to django.
Developers will need to run:
./tools/postgres-init-test-db
./tools/do-destroy-rebuild-test-database
this is assuming that they have already run:
./tools/postgres-init-db
./tools/do-destroy-rebuild-database
at some point on this pg_cluster. (The ordering is important; it will other-
wise complain about the south_migration table).
(imported from commit c56c6f27e13df7ae10b2e643e65d669dde61af3d)
This change will make it so that processes related to the app.humbughq.com
server are run under supervisord, which uses a state machine model to ensure
that programs are running. It also ensure process startup order.
We will need to manually switch the old way of running server (in screen) into
this new way of doing things, on both staging and prod (app_frontend.pp has been
updated appropriately). This means:
1) cp servers/puppet/modules/humbug/files/supervisord/conf.d/humbug.conf /etc/supervisord/conf.d
2) installing the supervisor package.
3) killing those while loops in that screen session
4) mkdir /var/log/humbug (as root)
5) /etc/init.d/supervisord start
6) check that nothing broke
(imported from commit 055269a70973db89acd69049e01b185fabdc8f90)
Most of the model logic pertaining to unread counts had been in
zephyr.js, along with a couple global variables. Now the code
is encapsulated in unread.js. It was a pretty straightforward
extraction with some minor method name changes. Also, a small
bit of the logic had also been in stream_list.js.
Conflicts:
tools/jslint/check-all.js
(imported from commit f0abdd48f26ab20c5beaef203479eb5a70dacfff)
The only known outstanding bug with this is that it doesn't properly
handle the updating of a message's highlighting/presence in a narrowed
view (e.g. in theory, a message should disappear if it is edited such
that its subject doesn't match your narrow or it no longer matches
your search). I think I'll just open a trac ticket about that once
this is merged, since it's a little hairy to deal with and kinda a
marginal use case.
Also it's not pretty, but that should be easy to tweak once we get the
framework merged.
Conflicts:
tools/jslint/check-all.js
(imported from commit 2d0e3a440bcd885546bd8e28aff97bf379649950)
South doesn't properly deal with removing the Django User model, so
this commit redoes our South history to instead start after that
migration has already been applied. This allows us to get rid of some
annoying hacks.
Note that developers and staging will need to run
./manage.py migrate --delete-ghost-migrations zephyr
in order to clear out the old versions of the migrations.
(imported from commit 7f45ea601b809dde33720f76e7dfb0ab348b0e65)
This is a prefactoring to eventually eliminate the home_unread_messages
global variable. More commits to follow.
In order to set up process_loaded_for_unread() not to modify
global variable to get its job done, we want to pull it out of
add_messages(), so that add_messages() doesn't have to pass back
state to the 9 different places in the codebase where it's called.
There are only 2 places where process_loaded_for_unread() get
called after this commit.
In order to facilitate pulling up process_loaded_for_unread(), I
made it so that the contract for add_messages() was to accept
already-hydrated messages. This way I could hydrate the messages
before calling process_loaded_for_unread() without have to
worry about double-caching them in add_messages. This will
slightly improve performance, but it was mostly done for code
clarity.
(imported from commit ad5aaad5b1f22c31647370f4c9dcb5f89d7d99a7)
The default is 50, which we believe was causing too much CPU
contention in prod during restarts. This commit lowers the max to 20.
The django-command-line script needs to be restarted manually to pick
up this change.
(imported from commit c2d097d2141be50b6222efb4a34142108c241cce)
The String.localeCompare function is really slow, at least partially
because it creates a locale-aware collator object each time. So now,
when we can, we create and cache a locale-aware collator object.
However, this is not supported on most browsers, so we fall back to a
non-locale-aware comparison. This is not ideal, but for now we are
mostly working with English-speaking customers.
(imported from commit 51aa02e3b9fe4a0ef0cb084874fe26e91c57f65e)
The previous version ended up being (at least sometimes) wrong after
the recent deployment system changes.
(imported from commit dec3beb1b1bf8b9c9ad6820b93b0a5d730d020e8)
This requires manual steps on deploy to each of staging and prod:
(1) Run the new update-deployment code to setup the initial deployment directory.
(2) Restart all the programs running in screen sessions.
(3) Deploy the nginx changes and restart nginx.
(imported from commit 1ffe27933ee79274dc0a93d35c9938712de0ef36)
This is a V1 of this feature. For now, the only way to expand is by narrowing
to the stream---future revisions may add a manual toggle if it is found to be
useful.
Additionally, showing per-subject unread counts will be coming in a future revision
as well.
(imported from commit fb5df0d27e928fa3b0f32b9ff2c1c508202cf7e5)
Django's South migrations support for setting up a new database
doesn't properly handle AUTH_USER_MODEL changing over time. Fix this
by having the initial migration be run with AUTH_USER_MODEL set to the
default value.
(imported from commit c373db9edc61f26527c486c741f8e870614600e3)
We accidentally lost this when we did the User/UserProfile merge (this
commit also deletes the old code to add the auth_user index in
do-destroy-rebuild-database).
This below is mostly just notes for future reference, but when
deploying this change to staging, we should consider running the
following instead of using the migration directly:
CREATE UNIQUE INDEX CONCURRENTLY zephyr_userprofile_email_uniq ON zephyr_userprofile(email);
ALTER TABLE zephyr_userprofile ADD CONSTRAINT zephyr_userprofile_email_uniq UNIQUE USING INDEX zephyr_userprofile_email_uniq;
CREATE INDEX CONCURRENTLY zephyr_userprofile_email ON zephyr_userprofile(email);
But I think it might be the case that it's fine to just run it
directly, since the ALTER TABLE part seems to hang if there's an open
transaction working on a UserProfile object anyway.
(imported from commit 1bf34ce242de51e97c91c8bab86b6b273e17fb43)
This allows blueslip to catch exceptions from the event handlers on
these elements in addition to the other benefits that not using
inline handlers provide.
(imported from commit 2bdcb2496c6c08fa7228a20ce6164b527cf64e41)
When the backend is failing, the frontend often fails in much less
clear ways (e.g. timing out), so it's generally more useful to run the
backend tests first.
(imported from commit 36ac862ad1dbb21e32c0f44ba135c3c29bbea2f5)
It doesn't save much time (maybe 0.5s out of 12+s).
I'm leaving the option in because I think it is still useful for
iteratively testing a single test case.
(imported from commit a0ac43f4c48eec101f05d731740394b30a15773b)
...rather than embedding them into index.html.
This is only acceptable for dev, but the next commit adds an alternative
mechanism for prod.
There isn't actually a manual deployment step here. However, this commit won't
work on staging / prod without the next one (since we don't serve
zephyr/static/templates in prod).
(imported from commit dce7ddfe89e07afc3a96699bb972fd124335aa05)
Before this commit, if you try to arrow around when the selected
message is outside the pointer threshold for recentering, you get a
big jump, even if you are arrowing towards the center of the viewport.
(imported from commit 5c15d5ccccdf027a8bfa8b79bf519fccbfa971d8)
The new nginx configuration file needs to be copied to
/etc/nginx/humbug-include and nginx needs to be restarted when this
commit is deployed.
(imported from commit 6c43f3c2c7a6acee6a852c672c96a38bda01dd0d)
This means all GET parameters were chopped off previously in local testing.
Nginx does pattern matching so this is not a problem on staging.
(imported from commit 25a28155b70d168228ca793fc0122b2ebea408e9)
If the user scrolls super fast, our scroll handler might not catch
the user passing by some messages.
(imported from commit 14cebffcd1321f02443971ac5e1c922db19648ab)
restart-server has been relatively slow recently, and it'd be nice to
know what it is spending its time doing when it hangs for a few
seconds.
(imported from commit a411c951f5a3f2a1366b6d5d3a40d0660ebec11b)
Our previous code could in theory end up clearing the caches it had
just filled, if Tornado's cache filling work happened to be faster
than the memcached flush.
(imported from commit 48174aadad398fb7a7c917a1df765c1261b12a55)
We were previously having an issue where the tutorial could
be pre-empted if you got a few messages while you were first
logging in.
I have some reservations about this being slightly fragile, and a
better approach might be to just have a bit that we use to determine
whether or not you've already seen a tutorial. (Or potentially that
checks whether or not you've ever sent a message.)
(imported from commit f8858f64a36bcd25887b76314caff283929f340c)
The new system, called blueslip, makes errors fatal when in debug
mode and only output a message when running in production. In the
future, it could also send user errors back to us automatically.
(imported from commit 1232607c0311e885c8b5a5e8a45ffb28822426e0)
This should substantially improve the repeat-rendering time for pages
with large numbers of tweets since we don't need to go all the way to
twitter.com, which can take like a second, to render tweets properly.
To deploy this commit properly, one needs to run
./manage.py createcachetable third_party_api_results
(imported from commit 01b528e61f9dde2ee718bdec0490088907b6017e)
By splitting up all_msg_list and home_msg_list, we can properly add/remove
streams from the home view without having to jump through hoops.
(imported from commit 92767197759f7519197dfc58be951b60fa823fbb)
Messages are now selected on a MessageList, which triggers a
message_selected event that other parts of the code can listen for.
(imported from commit 1da9e4121425c0ac4461b41b7aea169072e1512b)
Previously we would select the first message in the block. Now, we
only do that if a message that is selected will not be in the
resulting narrowed view. If the selected message will be in the
narrowed view, we select that message once narrowed.
(imported from commit 4da5a3a0b597b58c2e028f1b29ac20ae3808a4d1)
- Find Python and JavaScript files the same way, by looking at
everything in Git with specific exceptions
- Check the non-browser JavaScript, with appropriate JSLint options
Fixes#19 and #387.
(imported from commit 042bd6c26aa7a7ea2209935320ef4da12344b02b)
We leave the stuff under api/ alone for now, since we need to be able to ship
it as a standalone thing.
tools/post-receive wasn't using the function anyway.
For push to master: Push this commit, update post-receive per instructions at
the top of that file, then push the rest of the branch to confirm that the hook
still works.
No manual instructions for prod.
(imported from commit 9bcbe14c08d15eda47d82f0b702bad33e217a074)
At Ksplice we used /usr/bin/python because we shipped dependencies as Debian /
Red Hat packages, which would be installed against the system Python. We were
also very careful to use only Python 2.3 features so that even old system
Python would still work.
None of that is true at Humbug. We expect users to install dependencies
themselves, so it's more likely that the Python in $PATH is correct. On OS X
in particular, it's common to have five broken Python installs and there's no
expectation that /usr/bin/python is the right one.
The files which aren't marked executable are not interesting to run as scripts,
so we just remove the line there. (In general it's common to have libraries
that can also be executed, to run test cases or whatever, but that's not the
case here.)
(imported from commit 437d4aee2c6e66601ad3334eefd50749cce2eca6)
This commit does the actual work of formatting recent dates as weekday
names and updating the format when the date is sufficiently far in the
past.
Also, silence a bogus jslint warning.
(imported from commit 503286fbd9c1e33a81cc7d78cf8d08d5e7f78c1a)
This commit just moves time rendering logic to its own file, and does
not make any functionality changes.
(imported from commit d111d03c6abc8d9550fcf65e4f89eab8056d1ed4)
This code is no longer needed now that we send the Humbug _before_
restarting the server.
(imported from commit c281ac7db55ad2801d2da43d4aca5583ad48de93)
So here's the reproduction recipe:
(1) Find a narrow that doesn't have any messages since 4 days ago
(2) Directly visit that narrow in your browser (or wait for someone to
do a deploy and thus auto-reload)
(3) Wait until load_old_messages has been called at least once
(4) Un-narrow
(5) Scroll up, and notice that the 400 most recent messages are above
sets of older messages.
The cause is that the code in add_messages assumed that
selected_message_id was within the range of message already in the
home view. This is true in most cases where add_message is being
called, but it is not true in the case that the user was in a narrowed
view containing only very old messages (And thus selected_message_id
would be older than everything in the home view).
We can fix this by tracking persistent_message_id separately and using
that for the relevant test in add_messages.
(imported from commit f0da2561ba68f729343b260adc398029fae6acf7)
Deployment steps: See comment at the top of tools/post-receive.
Will do this when the commit hits master.
(imported from commit 8e096609fca618ed82bfbee43bae701daaf49261)
This was removed in 63788aa3. It was the only non-symlink in
zephyr/static-access-control/.
(imported from commit c9ee043b07c5fd4050aeaafa40394a6b69915bfb)
This meant that (in some circumstances, anyway) the script would bail after the
inner process was killed by restart-server.
Manual deployment steps:
After pushing, log into {staging,app} and restart these commands within
the screen session.
(imported from commit 696bc8148fa9df4b536253d54c65b7c4cb908530)
This now allows e.g.
./tools/test-backend zephyr.BugdownTest
or
./tools/test-backend zephyr.BugdownTest --verbosity=2
but unfortunately not
./tools/test-backend --verbosity=2 zephyr.BugdownTest
(imported from commit 95302db3efe45182d789c40c2a2899230a06b091)
Manual deployment steps: The same Nginx reload as for "Get rid of the
static-access-control mechanism". If deploying both commits at once,
just do it once.
(imported from commit dd8dbbf14b95fce0a4b6f66f462fa0a6b50bfb8c)
This way you can just extract it to a directory on your PYTHONPATH and
do an "import humbug"
(imported from commit d71de15fa51a0dde6534046902cfa058064a77ef)
If you create a spinner in a hidden element and then show the
element, the spinner is placed differently than if you had created
the spinner while the element was visible. This commit makes it so
that we never create spinners while their parent is hidden.
(imported from commit a21e68976d70fcceece30ee35f5e7cf6f9490497)
/?lurk=foo will show all messages to the stream "foo", regardless of
whether you're subscribed.
(imported from commit 049d98b3ee8df19ef0a9dc392ae941dd463f8dd5)
This tool hardcodes the paths to the input log files; we'll need to
change that down the line.
(imported from commit 8b067a6d3dc781694a001d549e5e60900d9c2596)
feedback-bot and zephyr_mirror will need to be updated and restarted
when this is deployed to prod.
(imported from commit fe2b524424c174bcb1b717a851a5d3815fda3f69)
This restores the time-travel functionality and fixes Waseem's laundry
list of problems with its original UI.
(imported from commit e30e02c25af994435adb815d26284b3669c945a4)
This uses pylibmc, the same memcached client library we use from Django. If
that becomes a problem for whatever reason, all we really need to do is send
"flush_all\n" to TCP port 11211 on localhost.
(imported from commit 0b9736bd31b0549b5dabd4b735706351635a9cf2)
We can't rely on the builtin exception printing, because killpg will kill the
process before that happens.
(imported from commit e7db6a39e0549e21653b4a705d2278eb3dc7b2d4)
This commit changes APIs and requires and update of all zephyr
mirroring bots to deploy properly.
(imported from commit 2672d2d07269379f7a865644aaeb6796d54183e1)
Personals are now just private messages between two people (which
sometimes manifests as a private message with one recipient). The
new message type on the send path is 'private'. Note that the receive
path still has 'personal' and 'huddle' message types.
(imported from commit 97a438ef5c0b3db4eb3e6db674ea38a081265dd3)
After this commit, we're tracking two branches: "master" and "prod".
Pushed to "prod" deploy to production, while pushes to "master" deploy
to the staging server.
(imported from commit 6ce429a1d6f606fb6136341dc393d93fd1228a21)
I think it might be nice to have it also specify some information
about the new review request, but I think we benefit a lot from
getting this minimal version out the door.
(imported from commit cb04c72ee125763cff2d83a6afad3f5319d8ebc6)
This also cleans up the autocomplete source specifications,
making the three typeaheads all look fairly consistent.
(imported from commit e72655d715db74cfc9ab45b51e7e2ff9e8ea84c5)
It's possible that the three places we can kill a popover
(body click, Esc, clicking on a new one) should be
refactored to use the same code.
(imported from commit ba7eab480fd2258abfb469c8f1155f29bc63f7f6)
Eventually this should go to the staging server, and we'll have a separate
process to migrate changes from there to production.
(imported from commit 2a712758844524fdf2f23f798baf6b607d056b9a)
Instead we infer this from narrow.active(), with the ability to override during
the narrowing procedure.
(imported from commit fab9c6861f19aedf0ee8af094c1ef4e8a0a73d80)
For reasons I don't totally understand, the previous code took like 2
seconds to compute the list of Python files.
(imported from commit 23668e7d79f1d69e1225389be7d8c8a3fbb6fad1)
We need this for client auto-reload on static file changes, because the server
generation comes from get_updates.
(imported from commit c5e97421883383c12891fdcb177d0f6fda4b4857)
Embedding this in index.html won't work anymore, because the Django FastCGI and
the Tornado servers might have been started at different times.
(imported from commit 187909d0593449cf2989857671f9ca526723e451)
Tim Abbott
Anders Kaseorg
Leo Franchi
Tim Abbott
Reid Barton
David Roe
Reid Barton
Cat Miller
Jonathan Dahan
Luke Faraone
Yoyo Zhou
Zev Benjamin
Jason Michalski
Waseem Daher
Tim Abbott
Steve Howell
Zev Benjamin
Jessica McKellar
Leo Franchi
acrefoot
Luke Faraone
Kevin Mehall
acrefoot
Jessica McKellar
Scott Feeney
Steve Howell
Tim Abbott
Kevin Mehall
Zev Benjamin
Luke Faraone
Scott Feeney
Leo Franchi
Keegan McAllister
Waseem Daher
Jeff Arnold
Reid Barton
Jacob Hurwitz