2024 年的期刊文章 · 第 37 页

共 1515 篇,按发表时间排,新的在前,每页 40 篇。这一页是 2024 到 2024。

按期刊、按主题看 →

第 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 页

JMLR Machine Learning Networks & Graphs Optimization & Learning Theory · 发表于 2024

Classification with Deep Neural Networks and Logistic Loss

Zihan Zhang, Lei Shi, Ding-Xuan Zhou

摘要

Deep neural networks (DNNs) trained with the logistic loss (also known as the cross entropy loss) have made impressive advancements in various binary classification tasks. Despite the considerable success in practice, generalization analysis for binary classification with deep neural networks and the logistic loss remains scarce. The unboundedness of the target function for the logistic loss in binary classification is the main obstacle to deriving satisfactory generalization bounds. In this paper, we aim to fill this gap by developing a novel theoretical analysis and using it to establish tight generalization bounds for training fully connected ReLU DNNs with logistic loss in binary classification. Our generalization analysis is based on an elegant oracle-type inequality which enables us to deal with the boundedness restriction of the target function. Using this oracle-type inequality, we establish generalization bounds for fully connected ReLU DNN classifiers $\hat{f}^{\text{FNN}}_n$ trained by empirical logistic risk minimization with respect to i.i.d. samples of size $n$, which lead to sharp rates of convergence as $n\to\infty$. In particular, we obtain optimal convergence rates for $\hat{f}^{\text{FNN}}_n$ (up to some logarithmic factor) only requiring the Hölder smoothness of the conditional class probability $\eta$ of data. Moreover, we consider a compositional assumption that requires $\eta$ to be the composition of several vector-valued multivariate functions of which each component function is either a maximum value function or a Hölder smooth function only depending on a small number of its input variables. Under this assumption, we can even derive optimal convergence rates for $\hat{f}^{\text{FNN}}_n$ (up to some logarithmic factor) which are independent of the input dimension of data. This result explains why in practice DNN classifiers can overcome the curse of dimensionality and perform well in high-dimensional classification problems. Furthermore, we establish dimension-free rates of convergence under other circumstances such as when the decision boundary is piecewise smooth and the input data are bounded away from it. Besides the novel oracle-type inequality, the sharp convergence rates presented in our paper also owe to a tight error bound for approximating the natural logarithm function near zero (where it is unbounded) by ReLU DNNs. In addition, we justify our claims for the optimality of rates by proving corresponding minimax lower bounds. All these results are new in the literature and will deepen our theoretical understanding of classification with deep neural networks.

JMLR Networks & Graphs Latent Variables & Mixtures · 发表于 2024

Classification of Data Generated by Gaussian Mixture Models Using Deep ReLU Networks

Tian-Yi Zhou, Xiaoming Huo

摘要

This paper studies the binary classification of unbounded data from ${\mathbb R}^d$ generated under Gaussian Mixture Models (GMMs) using deep ReLU neural networks. We obtain — for the first time — non-asymptotic upper bounds and convergence rates of the excess risk (excess misclassification error) for the classification without restrictions on model parameters. While the majority of existing generalization analysis of classification algorithms relies on a bounded domain, we consider an unbounded domain by leveraging the analyticity and fast decay of Gaussian distributions. To facilitate our analysis, we give a novel approximation error bound for general analytic functions using ReLU networks, which may be of independent interest. Gaussian distributions can be adopted nicely to model data arising in applications, e.g., speeches, images, and texts; our results provide a theoretical verification of the observed efficiency of deep neural networks in practical classification problems.

JMLR Other · 发表于 2024

Choosing the Number of Topics in LDA Models – A Monte Carlo Comparison of Selection Criteria

Victor Bystrov, Viktoriia Naboka-Krell, Anna Staszewska-Bystrova, Peter Winker

摘要

Selecting the number of topics in Latent Dirichlet Allocation (LDA) models is considered to be a difficult task, for which various approaches have been proposed. In this paper the performance of the recently developed singular Bayesian information criterion (sBIC) is evaluated and compared to the performance of alternative model selection criteria. The sBIC is a generalization of the standard BIC that can be applied to singular statistical models. The comparison is based on Monte Carlo simulations and carried out for several alternative settings, varying with respect to the number of topics, the number of documents and the size of documents in the corpora. Performance is measured using different criteria which take into account the correct number of topics, but also whether the relevant topics from the considered data generation processes (DGPs) are revealed. Practical recommendations for LDA model selection in applications are derived.

JMLR Optimization & Learning Theory Nonparametric & Semiparametric · 发表于 2024

Characterization of translation invariant MMD on Rd and connections with Wasserstein distances

Thibault Modeste, Clément Dombry

摘要

Kernel mean embeddings and maximum mean discrepancies (MMD) associated with positive definite kernels are important tools in machine learning that allow to compare probability measures and sample distributions. We provide a full characterization of translation invariant MMDs on $\mathbb{R}^d$ that are parametrized by a spectral measure and a semi-definite positive symmetric matrix. Furthermore, we investigate the connections between translation invariant MMDs and Wasserstein distances on $\mathbb{R}^d$. We show in particular that convergence with respect to the MMD associated with the Energy Kernel of order $\alpha\in(0,1)$ implies convergence with respect to the Wasserstein distance of order $\beta<\alpha$. We also provide examples of kernels metrizing the Wasserstein space of order $\alpha\geq 1$. A short numerical experiment illustrates our findings in the framework of the one-sample-test.

Econometrica Other · 发表于 2024

Certification Design With Common Values

Andreas Asseyer, Ran Weksler

摘要

This paper studies certification design and its implications for information disclosure. Our model features a profit‐maximizing certifier and the seller of a good of unknown quality. We allow for common values as the seller's opportunity cost may depend on the quality of the good. We compare certifier‐optimal with transparency‐maximizing certification design. Certifier‐optimal certification design implements the evidence structure of Dye (1985)—a fraction of sellers acquire information while the remaining sellers are uninformed—and results in partial disclosure to the market. A transparency‐maximizing regulator prefers a less precise signal, which conveys more information to the market through a higher rate of certification and unraveling (Grossman (1981), Milgrom (1981)) at the disclosure stage.

Econometrica Econometrics & Economics · 发表于 2024

Caution and Reference Effects

Simone Cerreia-Vioglio, David Dillenberger, Pietro Ortoleva

摘要

We introduce Cautious Utility, a new model based on the idea that individuals are unsure of trade‐offs between goods and apply caution. The model yields an endowment effect, even when gains and losses are treated symmetrically. Moreover, it implies either loss aversion or loss neutrality for risk, but in a way unrelated to the endowment effect, and it captures the certainty effect, providing a novel unified explanation of all three phenomena. Cautious Utility can help organize empirical evidence, including some that directly contradicts leading alternatives.

JMLR Causal Inference · 发表于 2024

Causal-learn: Causal Discovery in Python

Yujia Zheng, Biwei Huang, Wei Chen, Joseph Ramsey, Mingming Gong, Ruichu Cai, Shohei Shimizu, Peter Spirtes, Kun Zhang

摘要

Causal discovery aims at revealing causal relations from observational data, which is a fundamental task in science and engineering. We describe causal-learn, an open-source Python library for causal discovery. This library focuses on bringing a comprehensive collection of causal discovery methods to both practitioners and researchers. It provides easy-to-use APIs for non-specialists, modular building blocks for developers, detailed documentation for learners, and comprehensive methods for all. Different from previous packages in R or Java, causal-learn is fully developed in Python, which could be more in tune with the recent preference shift in programming languages within related communities. The library is available at https://github.com/py-why/causal-learn.

JMLR Causal Inference Clinical Trials · 发表于 2024

Causal effects of intervening variables in settings with unmeasured confounding

Lan Wen, Aaron Sarvet, Mats Stensrud

摘要

We present new results on average causal effects in settings with unmeasured exposure-outcome confounding. Our results are motivated by a class of estimands, e.g., frequently of interest in medicine and public health, that are currently not targeted by standard approaches for average causal effects. We recognize these estimands as queries about the average causal effect of an intervening variable. We anchor our introduction of these estimands in an investigation of the role of chronic pain and opioid prescription patterns, and illustrate how conventional approaches will lead to non-replicable estimates with ambiguous policy implications. We argue that our alternative effects are replicable and have clear policy implications, and furthermore are non-parametrically identified by the classical frontdoor formula. As an independent contribution, we derive a new semiparametric efficient estimator of the frontdoor formula with a uniform sample boundedness guarantee. This property is unique among previously-described estimators in its class, and we demonstrate superior performance in finite-sample settings. The theoretical results are applied to data from the National Health and Nutrition Examination Survey.

JMLR Causal Inference · 发表于 2024

Causal Discovery with Generalized Linear Models through Peeling Algorithms

Minjie Wang, Xiaotong Shen, Wei Pan

摘要

This article presents a novel method for causal discovery with generalized structural equation models suited for analyzing diverse types of outcomes, including discrete, continuous, and mixed data. Causal discovery often faces challenges due to unmeasured confounders that hinder the identification of causal relationships. The proposed approach addresses this issue by developing two peeling algorithms (bottom-up and top-down) to ascertain causal relationships and valid instruments. This approach first reconstructs a super-graph to represent ancestral relationships between variables, using a peeling algorithm based on nodewise GLM regressions that exploit relationships between primary and instrumental variables. Then, it estimates parent-child effects from the ancestral relationships using another peeling algorithm while deconfounding a child's model with information borrowed from its parents' models. The article offers a theoretical analysis of the proposed approach, establishing conditions for model identifiability and providing statistical guarantees for accurately discovering parent-child relationships via the peeling algorithms. Furthermore, the article presents numerical experiments showcasing the effectiveness of our approach in comparison to state-of-the-art structure learning methods without confounders. Lastly, it demonstrates an application to Alzheimer's disease (AD), highlighting the method's utility in constructing gene-to-gene and gene-to-disease regulatory networks involving Single Nucleotide Polymorphisms (SNPs) for healthy and AD subjects.

Econometrica Other · 发表于 2024

Can Restorative Justice Conferencing Reduce Recidivism? Evidence From the Make‐it‐Right Program

Yotam Shem-Tov, Steven Raphael, Alissa Skog

摘要

This paper studies the effect of a restorative justice intervention targeted at 143 youth ages 13 to 17 facing felony charges of medium severity (e.g., burglary, assault). Eligible youths were randomly assigned to participate in the Make‐it‐Right (MIR) restorative justice program or a control group where they faced standard criminal prosecution. We estimate the effects of MIR on the likelihood that a youth will be rearrested in the four years following randomization. Assignment to MIR reduces the probability of a rearrest within six months by 19 percentage points, a 44 percent reduction relative to the control group. Moreover, the reduction in recidivism persists even four years after randomization. Thus, our estimates show that restorative justice conferencing can reduce recidivism among youth charged with relatively serious offenses and can be an effective alternative to traditional criminal justice practices.

Econometrica Econometrics & Economics · 发表于 2024

Can Deficits Finance Themselves?

George-Marios Angeletos, Chen Lian, Christian K. Wolf

摘要

We ask how fiscal deficits are financed in environments with two key features: (i) nominal rigidity, and (ii) a violation of Ricardian equivalence due to finite lives or liquidity constraints. In such environments, deficits can contribute to their own financing through two channels: a boom in real economic activity, which expands the tax base; and a surge in inflation, which erodes the real value of nominal government debt. Our main theoretical result establishes that this mechanism becomes more potent as fiscal adjustment is delayed, leading to full self‐financing in the limit: if the monetary authority does not lean too heavily against the fiscal stimulus, then the government can run a deficit today, refrain from tax hikes or spending cuts in the future, and still see its debt converge back to its initial level. We further demonstrate that a significant degree of self‐financing is achievable when the theory is disciplined by empirical evidence on marginal propensities to consume, nominal rigidities, the monetary policy reaction, and the speed of fiscal adjustment.

JMLR Optimization & Learning Theory Machine Learning · 发表于 2024

Bridging Distributional and Risk-sensitive Reinforcement Learning with Provable Regret Bounds

Hao Liang, Zhi-Quan Luo

摘要

We study the regret guarantee for risk-sensitive reinforcement learning (RSRL) via distributional reinforcement learning (DRL) methods. In particular, we consider finite episodic Markov decision processes whose objective is the entropic risk measure (EntRM) of return. By leveraging a key property of the EntRM, the independence property, we establish the risk-sensitive distributional dynamic programming framework. We then propose two novel DRL algorithms that implement optimism through two different schemes, including a model-free one and a model-based one. We prove that they both attain $\tilde{\mathcal{O}}\left(\frac{\exp(|\beta| H)-1}{|\beta|}H\sqrt{S^2AK}\right)$ regret upper bound, where $S$, $A$, $K$, $H$, $T=KH$, and $\beta$ represent the number of states, actions, episodes, time horizon, number of total time-steps, and risk parameter respectively. It matches RSVI2, with novel distributional analysis that focuses on the distributions of returns rather than the risk values associated with these returns. To the best of our knowledge, this is the first regret analysis that bridges DRL and RSRL in terms of sample complexity. To address the computational inefficiencies inherent in the model-free DRL algorithm, we propose an alternative DRL algorithm with distribution representation. This approach effectively represents any bounded distribution using a refined distribution class. It significantly amplifies computational efficiency while maintaining the established regret bounds. We also prove a tighter minimax lower bound of $\Omega\left(\frac{\exp(\beta H/6)-1}{\beta }\sqrt{SAT}\right)$ for the $\beta>0$ case, which recovers the tight lower bound $\Omega(H\sqrt{SAT})$ in the risk-neutral setting.

JMLR Machine Learning · 发表于 2024

Boundary constrained Gaussian processes for robust physics-informed machine learning of linear partial differential equations

David Dalton, Alan Lazarus, Hao Gao, Dirk Husmeier

摘要

We introduce a framework for designing boundary constrained Gaussian process (BCGP) priors for exact enforcement of linear boundary conditions, and apply it to the machine learning of (initial) boundary value problems involving linear partial differential equations (PDEs).In contrast to existing work, we illustrate how to design boundary constrained mean and kernel functions for all classes of boundary conditions typically used in PDE modelling, namely Dirichlet, Neumann, Robin and mixed conditions. Importantly, this is done in a manner which allows for both forward and inverse problems to be naturally accommodated. We prove that the BCGP kernel has a universal representational capacity under Dirichlet conditions, and establish a formal equivalence between BCGPs and boundary-constrained neural networks (BCNNs) of infinite width.Finally, extensive numerical experiments are performed involving several linear PDEs, the results of which demonstrate the effectiveness and robustness of BCGP inference in the presence of sparse, noisy data.

Econometrica Testing & Uncertainty · 发表于 2024

Bootstrap Inference for Fixed‐Effect Models

Ayden Higgins, Koen Jochmans

摘要

The maximum‐likelihood estimator of nonlinear panel data models with fixed effects is asymptotically biased under rectangular‐array asymptotics. The literature has devoted substantial effort to devising methods that correct for this bias as a means to salvage standard inferential procedures. The chief purpose of this paper is to show that the (recursive, parametric) bootstrap replicates the asymptotic distribution of the (uncorrected) maximum‐likelihood estimator and of the likelihood‐ratio statistic. This justifies the use of confidence sets and decision rules for hypothesis testing constructed via conventional bootstrap methods. No modification for the presence of bias needs to be made.

JMLR Econometrics & Economics · 发表于 2024

Blessings and Curses of Covariate Shifts: Adversarial Learning Dynamics, Directional Convergence, and Equilibria

Tengyuan Liang

摘要

Covariate distribution shifts and adversarial perturbations present robustness challenges to the conventional statistical learning framework: mild shifts in the test covariate distribution can significantly affect the performance of the statistical model learned based on the training distribution. The model performance typically deteriorates when extrapolation happens: namely, covariates shift to a region where the training distribution is scarce, and naturally, the learned model has little information. For robustness and regularization considerations, adversarial perturbation techniques are proposed as a remedy; however, careful study needs to be carried out about what extrapolation region adversarial covariate shift will focus on, given a learned model. This paper precisely characterizes the extrapolation region, examining both regression and classification in an infinite-dimensional setting. We study the implications of adversarial covariate shifts to subsequent learning of the equilibrium---the Bayes optimal model---in a sequential game framework. We exploit the dynamics of the adversarial learning game and reveal the curious effects of the covariate shift to equilibrium learning and experimental design. In particular, we establish two directional convergence results that exhibit distinctive phenomena: (1) a blessing in regression, the adversarial covariate shifts in an exponential rate to an optimal experimental design for rapid subsequent learning; (2) a curse in classification, the adversarial covariate shifts in a subquadratic rate to the hardest experimental design trapping subsequent learning.

JMLR Bayesian Methods · 发表于 2024

Black Box Variational Inference with a Deterministic Objective: Faster, More Accurate, and Even More Black Box

Ryan Giordano, Martin Ingram, Tamara Broderick

摘要

Automatic differentiation variational inference (ADVI) offers fast and easy-to-use posterior approximation in multiple modern probabilistic programming languages. However, its stochastic optimizer lacks clear convergence criteria and requires tuning parameters. Moreover, ADVI inherits the poor posterior uncertainty estimates of mean-field variational Bayes (MFVB). We introduce "deterministic ADVI" (DADVI) to address these issues. DADVI replaces the intractable MFVB objective with a fixed Monte Carlo approximation, a technique known in the stochastic optimization literature as the "sample average approximation" (SAA). By optimizing an approximate but deterministic objective, DADVI can use off-the-shelf second-order optimization, and, unlike standard mean-field ADVI, is amenable to more accurate posterior covariances via linear response (LR). In contrast to existing worst-case theory, we show that, on certain classes of common statistical problems, DADVI and the SAA can perform well with relatively few samples even in very high dimensions, though we also show that such favorable results cannot extend to variational approximations that are too expressive relative to mean-field ADVI. We show on a variety of real-world problems that DADVI reliably finds good solutions with default settings (unlike ADVI) and, together with LR covariances, is typically faster and more accurate than standard ADVI.

Econometrica Testing & Uncertainty · 发表于 2024

Bias‐Aware Inference in Fuzzy Regression Discontinuity Designs

Claudia Noack, Christoph Rothe

摘要

We propose new confidence sets (CSs) for the regression discontinuity parameter in fuzzy designs. Our CSs are based on local linear regression, and are bias‐aware, in the sense that they take possible bias explicitly into account. Their construction shares similarities with that of Anderson–Rubin CSs in exactly identified instrumental variable models, and thereby avoids issues with “delta method” approximations that underlie most commonly used existing inference methods for fuzzy regression discontinuity analysis. Our CSs are asymptotically equivalent to existing procedures in canonical settings with strong identification and a continuous running variable. However, they are also valid under a wide range of other empirically relevant conditions, such as setups with discrete running variables, donut designs, and weak identification.

Econometrica Other · 发表于 2024

Beyond Unbounded Beliefs: How Preferences and Information Interplay in Social Learning

Navin Kartik, SangMok Lee, Tianhao Liu, Daniel Rappoport

摘要

When does society eventually learn the truth, or take the correct action, via observational learning? In a general model of sequential learning over social networks, we identify a simple condition for learning dubbed excludability . Excludability is a joint property of agents' preferences and their information. We develop two classes of preferences and information that jointly satisfy excludability: (i) for a one‐dimensional state, preferences with single‐crossing differences and a new informational condition, directionally unbounded beliefs; and (ii) for a multi‐dimensional state, intermediate preferences and subexponential location‐shift information. These applications exemplify that with multiple states, “unbounded beliefs” is not only unnecessary for learning, but incompatible with familiar informational structures like normal information. Unbounded beliefs demands that a single agent can identify the correct action. Excludability, on the other hand, only requires that a single agent must be able to displace any wrong action, even if she cannot take the correct action.

JMLR Machine Learning · 发表于 2024

BenchMARL: Benchmarking Multi-Agent Reinforcement Learning

Matteo Bettini, Amanda Prorok, Vincent Moens

摘要

The field of Multi-Agent Reinforcement Learning (MARL) is currently facing a reproducibility crisis. While solutions for standardized reporting have been proposed to address the issue, we still lack a benchmarking tool that enables standardization and reproducibility, while leveraging cutting-edge Reinforcement Learning (RL) implementations. In this paper, we introduce BenchMARL, the first MARL training library created to enable standardized benchmarking across different algorithms, models, and environments. BenchMARL uses TorchRL as its backend, granting it high-performance and maintained state-of-the-art implementations while addressing the broad community of MARL PyTorch users. Its design enables systematic configuration and reporting, thus allowing users to create and run complex benchmarks from simple one-line inputs. BenchMARL is open-sourced on GitHub at https://github.com/facebookresearch/BenchMARL

JMLR Bayesian Methods Networks & Graphs Genetics & Omics · 发表于 2024

Bayesian Structural Learning with Parametric Marginals for Count Data: An Application to Microbiota Systems

Veronica Vinciotti, Pariya Behrouzi, Reza Mohammadi

摘要

High dimensional and heterogeneous count data are collected in various applied fields. In this paper, we look closely at high-resolution sequencing data on the microbiome, which have enabled researchers to study the genomes of entire microbial communities. Revealing the underlying interactions between these communities is of vital importance to learn how microbes influence human health. To perform structural learning from multivariate count data such as these, we develop a novel Gaussian copula graphical model with two key elements. Firstly, we employ parametric regression to characterize the marginal distributions. This step is crucial for accommodating the impact of external covariates. Neglecting this adjustment could potentially introduce distortions in the inference of the underlying network of dependences. Secondly, we advance a Bayesian structure learning framework, based on a computationally efficient search algorithm that is suited to high dimensionality. The approach returns simultaneous inference of the marginal effects and of the dependence structure, including graph uncertainty estimates. A simulation study and a real data analysis of microbiome data highlight the applicability of the proposed approach at inferring networks from multivariate count data in general, and its relevance to microbiome analyses in particular. The proposed method is implemented in the R package BDgraph.

JMLR Econometrics & Economics Bayesian Methods · 发表于 2024

Bayesian Regression Markets

Thomas Falconer, Jalal Kazempour, Pierre Pinson

摘要

Although machine learning tasks are highly sensitive to the quality of input data, relevant datasets can often be challenging for firms to acquire, especially when held privately by a variety of owners. For instance, if these owners are competitors in a downstream market, they may be reluctant to share information. Focusing on supervised learning for regression tasks, we develop a regression market to provide a monetary incentive for data sharing. Our mechanism adopts a Bayesian framework, allowing us to consider a more general class of regression tasks. We present a thorough exploration of the market properties, and show that similar proposals in literature expose the market agents to sizeable financial risks, which can be mitigated in our setup.

Econometrica Other · 发表于 2024

Bargaining and Exclusion With Multiple Buyers

Dilip Abreu, Mihai Manea

摘要

A seller trades with q out of n buyers who have valuations a 1 ≥ a 2 ≥ ⋯ ≥ a n > 0 via sequential bilateral bargaining. When q < n, buyer payoffs vary across equilibria in the patient limit, but seller payoffs do not, and converge to maxl≤q+1[(a1+a2+⋯+al−1)/2+al+1+⋯+aq+1]. If l * is the (generically unique) maximizer of this optimization problem, then each buyer i < l * trades with probability 1 at the fair price a i /2, while buyers i ≥ l * are excluded from trade with positive probability. Bargaining with buyers who face the threat of exclusion is driven by a sequential outside option principle: the seller can sequentially exercise the outside option of trading with the extra marginal buyer q + 1, then with the new extra marginal buyer q, and so on, extracting full surplus from each buyer in this sequence and enhancing the outside option at every stage. A seller who can serve all buyers ( q = n) may benefit from creating scarcity by committing to exclude some remaining buyers as negotiations proceed. An optimal exclusion commitment, within a general class, excludes a single buyer but maintains flexibility about which buyer is excluded. Results apply symmetrically to a buyer bargaining with multiple sellers.

JMLR Other · 发表于 2024

Bagging Provides Assumption-free Stability

Jake A. Soloff, Rina Foygel Barber, Rebecca Willett

摘要

Bagging is an important technique for stabilizing machine learning models. In this paper, we derive a finite-sample guarantee on the stability of bagging for any model. Our result places no assumptions on the distribution of the data, on the properties of the base algorithm, or on the dimensionality of the covariates. Our guarantee applies to many variants of bagging and is optimal up to a constant. Empirical results validate our findings, showing that bagging successfully stabilizes even highly unstable base algorithms.

JMLR Machine Learning · 发表于 2024

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

BigScience Workshop

摘要

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.

JMLR Causal Inference · 发表于 2024

Axiomatic effect propagation in structural causal models

Raghav Singal, George Michailidis

摘要

We study effect propagation in a causal directed acyclic graph (DAG), with the goal of providing a flow-based decomposition of the effect (i.e., change in the outcome variable) as a result of changes in the source variables. We first compare various ideas on causality to quantify effect propagation, such as direct and indirect effects, path-specific effects, and degree of responsibility. We discuss the shortcomings of such approaches and propose a flow-based methodology, which we call recursive Shapley value (RSV). By considering a broader set of counterfactuals than existing methods, RSV obeys a unique adherence to four desirable flow-based axioms. Further, we provide a general path-based characterization of RSV for an arbitrary non-parametric structural equations model (SEM) defined on the underlying DAG. Interestingly, for the special class of linear SEMs, RSV exhibits a simple and tractable characterization (and hence, computation), which recovers the classical method of path coefficients and is equivalent to path-specific effects. For non-parametric SEMs, we use our general characterization to develop an unbiased Monte-Carlo estimation procedure with an exponentially decaying sample complexity. We showcase the application of RSV on two challenging problems on causality (causal overdetermination and causal unfairness).

Econometrica Other · 发表于 2024

Attributes: Selective Learning and Influence

Arjada Bardhi

摘要

An agent selectively samples attributes of a complex project so as to influence the decision of a principal. The players disagree about the weighting, or relevance, of attributes. The correlation across attributes is modeled through a Gaussian process, the covariance function of which captures pairwise attribute similarity. The key trade‐off in sampling is between the alignment of the players' posterior values for the project and the variability of the principal's decision. Under a natural property of the attribute correlation—the nearest‐attribute property (NAP)—each optimal attribute is relevant for some player and at most two optimal attributes are relevant for only one player. We derive comparative statics in the strength of attribute correlation and examine the robustness of our findings to violations of NAP for a tractable class of distance‐based covariances. The findings carry testable implications for attribute‐based product evaluation and strategic selection of pilot sites.

JMLR Causal Inference · 发表于 2024

Assessing the Overall and Partial Causal Well-Specification of Nonlinear Additive Noise Models

Christoph Schultheiss, Peter Bühlmann

摘要

We propose a method to detect model misspecifications in nonlinear causal additive and potentially heteroscedastic noise models. We aim to identify predictor variables for which we can infer the causal effect even in cases of such misspecification. We develop a general framework based on knowledge of the multivariate observational data distribution. We then propose an algorithm for finite sample data, discuss its asymptotic properties, and illustrate its performance on simulated and real data.

JMLR Other · 发表于 2024

Approximate Information Tests on Statistical Submanifolds

Michael W. Trosset, Carey E. Priebe

摘要

Parametric inference posits a statistical model that is a specified family of probability distributions. Restricted inference, for example, restricted likelihood ratio testing, attempts to exploit the structure of a statistical submodel that is a subset of the specified family. We consider the problem of testing a simple hypothesis against alternatives from such a submodel. In the case of an unknown submodel, it is not clear how to realize the benefits of restricted inference. To do so, we first construct information tests that are locally asymptotically equivalent to likelihood ratio tests. Information tests are conceptually appealing but (in general) computationally intractable. However, unlike restricted likelihood ratio tests, restricted information tests can be approximated even when the statistical submodel is unknown. We construct approximate information tests using manifold learning procedures to extract information from samples of an unknown (or intractable) submodel, thereby providing a roadmap for computational solutions to a class of previously impenetrable problems in statistical inference. Examples illustrate the efficacy of the proposed methodology.

JMLR Bayesian Methods · 发表于 2024

Approximate Bayesian inference from noisy likelihoods with Gaussian process emulated MCMC

Marko Järvenpää, Jukka Corander

摘要

We present a framework for approximate Bayesian inference intended for a situation where only a limited number of noisy log-likelihood evaluations can be obtained due to constraints on the available computational budget, which is becoming increasingly common for expensive simulator-based models. We model the log-likelihood function using a Gaussian process (GP) and our main methodological innovation is to apply this model to emulate the progression that an exact Metropolis-Hastings (MH) sampler would take if it was applicable. Informative log-likelihood evaluation locations are selected using a sequential experimental design strategy until the MH accept/reject decisions are performed with sufficient level of accuracy based on a prespecified error tolerance criterion. The resulting approximate sampler is conceptually simple and shown to be sample-efficient. It is also more robust compared with earlier “Bayesian optimisation-like” methods tailored for approximate Bayesian inference, which generally assume a global surrogate model across the parameter space that can be challenging to fit well. We discuss some theoretical aspects and various interpretations of the resulting approximate MH sampler, and demonstrate its benefits in the context of Bayesian and generalised Bayesian likelihood-free inference for simulator-based statistical models.

JMLR Optimization & Learning Theory · 发表于 2024

An Optimal Transport Approach for Computing Adversarial Training Lower Bounds in Multiclass Classification

Nicolas Garcia Trillos, Matt Jacobs, Jakwang Kim, Matthew Werenski

摘要

Despite the success of deep learning-based algorithms, it is widely known that neural networks may fail to be robust. A popular paradigm to enforce robustness is adversarial training (AT), however, this introduces many computational and theoretical difficulties. Recent works have developed a connection between AT in the multiclass classification setting and multimarginal optimal transport (MOT), unlocking a new set of tools to study this problem. In this paper, we leverage the MOT connection to propose computationally tractable numerical algorithms for computing universal lower bounds on the optimal adversarial risk and identifying optimal classifiers. We propose two main algorithms based on linear programming (LP) and entropic regularization (Sinkhorn). Our key insight is that one can harmlessly truncate the higher order interactions between classes, preventing the combinatorial run times typically encountered in MOT problems. We validate these results with experiments on MNIST and CIFAR-$10$, which demonstrate the tractability of our approach.

JMLR Other · 发表于 2024

An Inexact Projected Regularized Newton Method for Fused Zero-norms Regularization Problems

Yuqia Wu, Shaohua Pan, Xiaoqi Yang

摘要

This paper concerns structured $\ell_0$-norms regularization problems, with a twice continuously differentiable loss function and a box constraint. This class of problems have a wide range of applications in statistics, machine learning and image processing. To the best of our knowledge, there is no efficient algorithm in the literature for solving them. In this paper, we first provide a polynomial-time algorithm to find a point in the proximal mapping of the fused $\ell_0$-norms with a box constraint based on dynamic programming principle. We then propose a hybrid algorithm of proximal gradient method and inexact projected regularized Newton method to solve structured $\ell_0$-norms regularization problems. The iterate sequence generated by the algorithm is shown to be convergent by virtue of a non-degeneracy condition, a curvature condition and a Kurdyka-Łojasiewicz property. A superlinear convergence rate of the iterates is established under a locally Hölderian error bound condition on a second-order stationary point set, without requiring the local optimality of the limit point. Finally, numerical experiments are conducted to highlight the features of our considered model, and the superiority of our proposed algorithm.

JMLR Other · 发表于 2024

An Entropy-Based Model for Hierarchical Learning

Amir R. Asadi

摘要

Machine learning, the predominant approach in the field of artificial intelligence, enables computers to learn from data and experience. In the supervised learning framework, accurate and efficient learning of dependencies between data instances and their corresponding labels requires auxiliary information about the data distribution and the target function. This central concept aligns with the notion of regularization in statistical learning theory. Real-world datasets are often characterized by multiscale data instance distributions and well-behaved, smooth target functions. Scale-invariant probability distributions, such as power-law distributions, provide notable examples of multiscale data instance distributions in various contexts. This paper introduces a hierarchical learning model that leverages such a multiscale data structure with a multiscale entropy-based training procedure and explores its statistical and computational advantages. The hierarchical learning model is inspired by the logical progression in human learning from easy to complex tasks and features interpretable levels. In this model, the logarithm of any data instance’s norm can be construed as the data instance's complexity, and the allocation of computational resources is tailored to this complexity, resulting in benefits such as increased inference speed. Furthermore, our multiscale analysis of the statistical risk yields stronger guarantees compared to conventional uniform convergence bounds.

JMLR Other · 发表于 2024

An Embedding Framework for the Design and Analysis of Consistent Polyhedral Surrogates

Jessie Finocchiaro, Rafael M. Frongillo, Bo Waggoner

摘要

We formalize and study the natural approach of designing convex surrogate loss functions via embeddings, for discrete problems such as classification, ranking, or structured prediction. In this approach, one embeds each of the finitely many predictions (e.g. rankings) as a point in $\mathbb{R}^d$, assigns the original loss values to these points, and “convexifies” the loss in some way to obtain a surrogate. We establish a strong connection between this approach and polyhedral (piecewise-linear convex) surrogate losses: every discrete loss is embedded by some polyhedral loss, and every polyhedral loss embeds some discrete loss. Moreover, an embedding gives rise to a consistent link function as well as linear surrogate regret bounds. Our results are constructive, as we illustrate with several examples. In particular, our framework gives succinct proofs of consistency or inconsistency for existing polyhedral surrogates, and for inconsistent surrogates, it further reveals the discrete losses for which these surrogates are consistent. We go on to show additional structure of embeddings, such as the equivalence of embedding and matching Bayes risks, and the equivalence of various notions of non-redudancy. Using these results, we establish that indirect elicitation, a necessary condition for consistency, is also sufficient when working with polyhedral surrogates.

JMLR Other · 发表于 2024

An Asymptotic Study of Discriminant and Vote-Averaging Schemes for Randomly-Projected Linear Discriminants

Lama B. Niyazi, Abla Kammoun, Hayssam Dahrouj, Mohamed-Slim Alouini, Tareq Y. Al-Naffouri

摘要

Modern technology has contributed to the rise of high-dimensional data in various domains such as bio-informatics, chemometrics, and face recognition. In the recent literature, random projections and, in particular, randomly-projected ensembles based on the classical Linear Discriminant Analysis (LDA), have been proposed for classification problems involving such high-dimensional data. In this work, we study the two main classes of randomly-projected LDA ensemble classifiers, namely discriminant averaging and vote averaging. Through asymptotic analysis in a growth regime where the problem dimensions are assumed to grow at constant rates to each other for a fixed ensemble size, we determine the exact mechanism through which the ensemble size affects the classification performance. Furthermore, we investigate whether projection selection truly matters in an ensemble setting, and, ultimately, derive the optimal form of the randomly-projected LDA ensemble. Motivated by these findings, we propose a framework for efficient tuning of the optimal classifier's ensemble size and projection dimension based on an estimator of the classifier probability of misclassification which is consistent under the assumed growth regime. The proposed framework is shown to outperform the existing rule-of-thumb, as well as other methods for parameter tuning, on both real and synthetic data.

JMLR Machine Learning · 发表于 2024

An Analysis of Quantile Temporal-Difference Learning

Mark Rowland, Rémi Munos, Mohammad Gheshlaghi Azar, Yunhao Tang, Georg Ostrovski, Anna Harutyunyan, Karl Tuyls, Marc G. Bellemare, Will Dabney

摘要

We analyse quantile temporal-difference learning (QTD), a distributional reinforcement learning algorithm that has proven to be a key component in several successful large-scale applications of reinforcement learning. Despite these empirical successes, a theoretical understanding of QTD has proven elusive until now. Unlike classical TD learning, which can be analysed with standard stochastic approximation tools, QTD updates do not approximate contraction mappings, are highly non-linear, and may have multiple fixed points. The core result of this paper is a proof of convergence to the fixed points of a related family of dynamic programming procedures with probability 1, putting QTD on firm theoretical footing. The proof establishes connections between QTD and non-linear differential inclusions through stochastic approximation theory and non-smooth analysis.

JMLR Networks & Graphs Machine Learning Spatial & Time Series · 发表于 2024

An Algorithmic Framework for the Optimization of Deep Neural Networks Architectures and Hyperparameters

Julie Keisler, El-Ghazali Talbi, Sandra Claudel, Gilles Cabriel

摘要

In this paper, we propose DRAGON (for DiRected Acyclic Graph OptimizatioN), an algorithmic framework to automatically generate efficient deep neural networks architectures and optimize their associated hyperparameters. The framework is based on evolving Directed Acyclic Graphs (DAGs), defining a more flexible search space than the existing ones in the literature. It allows mixtures of different classical operations: convolutions, recurrences and dense layers, but also more newfangled operations such as self-attention. Based on this search space we propose neighbourhood and evolution search operators to optimize both the architecture and hyper-parameters of our networks. These search operators can be used with any metaheuristic capable of handling mixed search spaces. We tested our algorithmic framework with an asynchronous evolutionary algorithm on a time series forecasting benchmark. The results demonstrate that DRAGON outperforms state-of-the-art handcrafted models and AutoML techniques for time series forecasting on numerous datasets. DRAGON has been implemented as a python open-source package.

JMLR Optimization & Learning Theory · 发表于 2024

An Algorithm with Optimal Dimension-Dependence for Zero-Order Nonsmooth Nonconvex Stochastic Optimization

Guy Kornowski, Ohad Shamir

摘要

We study the complexity of producing $(\delta,\epsilon)$-stationary points of Lipschitz objectives which are possibly neither smooth nor convex, using only noisy function evaluations. Recent works proposed several stochastic zero-order algorithms that solve this task, all of which suffer from a dimension-dependence of $\Omega(d^{3/2})$ where $d$ is the dimension of the problem, which was conjectured to be optimal. We refute this conjecture by providing a faster algorithm that has complexity $O(d\delta^{-1}\epsilon^{-3})$, which is optimal (up to numerical constants) with respect to $d$ and also optimal with respect to the accuracy parameters $\delta,\epsilon$, thus solving an open question due to Lin et al. (2022). Moreover, the convergence rate achieved by our algorithm is also optimal for smooth objectives, proving that in the nonconvex stochastic zero-order setting, nonsmooth optimization is as easy as smooth optimization. We provide algorithms that achieve the aforementioned convergence rate in expectation as well as with high probability. Our analysis is based on a simple yet powerful lemma regarding the Goldstein-subdifferential set, which allows utilizing recent advancements in first-order nonsmooth nonconvex optimization.

Econometrica Other · 发表于 2024

Ambiguous Contracts

Paul Dütting, Michal Feldman, Daniel Peretz, Larry Samuelson

摘要

We explore the deliberate infusion of ambiguity into the design of contracts. We show that when the agent is ambiguity‐averse and hence chooses an action that maximizes their minimum utility, the principal can strictly gain from using an ambiguous contract, and this gain can be arbitrarily high. We characterize the structure of optimal ambiguous contracts, showing that ambiguity drives optimal contracts toward simplicity. We also provide a characterization of ambiguity‐proof classes of contracts, where the principal cannot gain by infusing ambiguity. Finally, we show that when the agent can engage in mixed actions, the advantages of ambiguous contracts disappear.

JMLR Optimization & Learning Theory · 发表于 2024

Almost Sure Convergence Rates Analysis and Saddle Avoidance of Stochastic Gradient Methods

Jun Liu, Ye Yuan

摘要

The vast majority of convergence rates analysis for stochastic gradient methods in the literature focus on convergence in expectation, whereas trajectory-wise almost sure convergence is clearly important to ensure that any instantiation of the stochastic algorithms would converge with probability one. Here we provide a unified almost sure convergence rates analysis for stochastic gradient descent (SGD), stochastic heavy-ball (SHB), and stochastic Nesterov's accelerated gradient (SNAG) methods. We show, for the first time, that the almost sure convergence rates obtained for these stochastic gradient methods on strongly convex functions, are arbitrarily close to their optimal convergence rates possible. For non-convex objective functions, we not only show that a weighted average of the squared gradient norms converges to zero almost surely, but also the last iterates of the algorithms. We further provide last-iterate almost sure convergence rates analysis for stochastic gradient methods on general convex smooth functions, in contrast with most existing results in the literature that only provide convergence in expectation for a weighted average of the iterates. The last-iterate almost sure convergence results also enable us to obtain almost sure avoidance of any strict saddle manifold by stochastic gradient methods with or without momentum. To the best of our knowledge, this is the first time such results are obtained for SHB and SNAG methods.

Econometrica Econometrics & Economics · 发表于 2024

Aggregate Implications of Barriers to Female Entrepreneurship

Gaurav Chiplunkar, Pinelopi Koujianou Goldberg

摘要

We develop a framework for quantifying barriers to labor force participation (LFP) and entrepreneurship faced by women in India. We find substantial barriers to LFP, and higher costs of expanding businesses through hiring workers for women entrepreneurs. However, there is one area where female entrepreneurs have an advantage: the hiring of female workers. We show that this is not driven by the sectoral composition of female employment. Consistent with this pattern, policies promoting female entrepreneurship can significantly increase female LFP even without explicitly targeting female LFP. Counterfactual simulations indicate that removing all excess barriers faced by women entrepreneurs would substantially increase the fraction of female‐owned firms, female LFP, earnings, and generate substantial gains for the economy. These gains are due to higher LFP, higher real wages and profits, and reallocation: low productivity male‐owned firms previously sheltered from female competition are replaced by higher productivity female‐owned firms previously excluded from the economy.

第 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 页